Every performance number in my READMEs is backed by committed benchmark output tagged with the device and the commit that produced it.
Voice Journal's README states two numbers: 6.31% word error rate with voice activity detection off, and 4.50% with it on. Those numbers are not marketing copy. They are the output of a benchmark that ran on a physical Pixel 10 Pro XL, and the raw results are committed to the repository as JSON, tagged with the device that produced them and the git SHA of the exact code that was measured.
That last part is the whole point of this post. Anyone can claim an accuracy number. The question is whether the claim is reproducible, and reproducibility requires answering three questions that most performance claims quietly skip: measured on what hardware, running which exact code, producing what raw output.
The standard I hold across all my projects is simple to state: nothing is done until it is verified, and a performance claim without committed evidence is not a claim, it is a hope.
In practice that means benchmark results live in the repo next to the code they describe. The JSON records the device and the commit hash, so six months from now, when the codebase has moved on, the claim is still anchored to something checkable. If I optimize the transcription pipeline and the numbers improve, the new numbers get a new benchmark run, a new JSON file, and a new SHA. The old claim never silently becomes a lie.
Hardware verification is a hard gate for me, not a checkbox, because the emulator will lie to you in both directions. It lies optimistically about anything touching real microphones, thermals, and memory pressure. And in one memorable case it lied pessimistically in a way that nearly sank the project.
Early Whisper benchmarks on-device were so slow the whole concept looked dead. Transcription that should have taken seconds took long enough that no one would ever use it. The culprit was not the model or the phone. The Android Gradle Plugin compiles native code at -O0 in debug builds, so I was benchmarking completely unoptimized C++ and treating the result as the truth about whisper.cpp on mobile. One line in CMakeLists, forcing -O3 with -DNDEBUG in debug configuration, changed the answer entirely.
The lesson cuts both ways. If I had trusted that first number, I would have abandoned a viable design. If I had shipped without measuring on hardware at all, I would have made claims I never actually tested. Verification is not just about honesty toward users. It is about not deceiving yourself.
The two WER numbers exist because they answer different questions. With voice activity detection off, the model transcribes everything, including silence and noise, which is the harsher and more honest stress test. With VAD on, silence is filtered before transcription, which is how the app actually runs and how users actually experience it. Reporting only the flattering number would be marketing. Reporting only the harsh one would misrepresent real-world behavior. So the JSON contains both, and the README explains the difference.
This standard costs time. Benchmark harnesses are unglamorous work, running them on a physical phone is slower than trusting the emulator, and committing evidence forces you to confront results you might prefer to round up. But the alternative is a portfolio of assertions, and assertions are worth exactly what they cost to make.
Every project I ship follows the same gate. Tests ship with the feature, never after it. Claims ship with evidence, never without it. If a number appears in one of my READMEs, there is a file in the repo that shows where it came from, on what device, at what commit. Done means verified, and nothing else counts.
Enjoyed this one? Say hi and let’s talk shop.
Get in Touch