WER is the wrong number.

We have spent the last few months measuring what happens to a spoken request between the microphone and the model. Three of those results do not fit the number the industry uses to score transcription, and they do not fit it in three different directions.

A homophone substitution — their for there, the name Bill for the noun bill — is a genuine recognition failure. The recogniser heard the wrong word and wrote it down. Downstream it costs 0.4 points of accuracy.¹

Stripping the fillers out of a transcript makes it measurably closer to a written reference. Every um removed is one fewer insertion to count. Downstream it recovers −0.22 ± 1.08 points, which is to say nothing at all.¹

Rewriting the same request into clean, concise written English produces a transcript that would pass any editor. Downstream it costs 24.1 points — the largest single effect in the entire suite, larger than any keyboard typo, any dropped word, any mangled number.¹

A metric that scores the free error as an error, the useless clean-up as an improvement, and the catastrophic rewrite as good prose is not measuring the thing you are building on. We think it should be replaced, and we think the replacement is straightforward.

What WER actually measures

Word Error Rate is edit distance over tokens, normalised by the length of the reference: substitutions plus deletions plus insertions, over the number of words someone said. It is a good metric with a clear lineage. It comes from dictation, where the transcript was the product, and where a wrong word on the page was the whole of the harm.

One property does all the damage when you move it downstream: every edit is weighted equally. Kim and colleagues put the problem in four words. Take the reference this is a cat, and two hypotheses: this is the cat and this is a cap. One substitution each, against a four-word reference. Identical error rate. One of them is still about a cat.²

This is not a contrived example. On a production system, the same authors report three sets of hypotheses sitting at an identical 7.44 word error rate whose named-entity F1 runs from 0.590 to 0.846.² A quarter of the entity accuracy moves while the metric does not move at all. If you were choosing between those three systems on WER, you would be choosing at random.

Which tokens survive

The mechanism is not mysterious, and it has been measured. In the study above, token survival correlates with accuracy loss at r = +0.79. The authors put it plainly: destroying the original tokens is what hurts, and adding new ones on top is close to free.¹

That asymmetry alone breaks WER, which weights an insertion and a deletion the same. But survival is not quite the whole story either, and the three results show why. Each one destroys tokens; they differ in what the destroyed tokens were carrying.

A homophone destroys a token and leaves its information intact, because there in that position is reconstructable from everything around it. A stripped filler destroys a token that was carrying nothing to begin with. A fluent rewrite destroys tokens whose information is not recoverable from what remains — the specific time, the named system, the condition attached to the instruction — and it does so while producing text that reads better than the original.

So the quantity that predicts the outcome is not how many tokens changed. It is whether the tokens the action depends on are still recoverable afterwards. WER counts the first and is blind to the second.

This is not an artefact of English, or of one scoring convention. In Korean spoken question answering, where character error rate is the house metric, Jung and Choi isolated 1,206 cases in which a single character came out wrong. 87.6% of them landed on content words rather than grammatical particles, and 151 of them — 12.5% — became outright semantic failures, where the gold answer was entirely absent from the model’s response.⁵ The other seven-eighths did not. Every one of those 1,206 errors is one character, and a character-level metric prices them identically.

The same paper reports something that ought to be more unsettling than it usually is. Across four language models with very different absolute accuracy — 0.663 to 0.870 F1 on clean transcripts — the relative degradation as the audio got noisier was nearly identical: about 99% of baseline at +20 dB, 96% at +5 dB, 67% at −10 dB.⁵ What the cascade loses is set at the transcription stage, not by the model you put downstream. A better reader does not buy you out of it.

Twenty-three years of knowing this

None of this is a new discovery, and we want to be careful not to present it as one. In 2003, Wang, Acero and Chelba built a recogniser whose word error rate was 46% higher than the trigram baseline, and which reduced slot understanding error by as much as 17%.³ Their conclusion was that the language model should be trained against the understanding objective rather than against the transcription one. That was twenty-three years ago, on an air-travel booking system.

The finding kept being rediscovered because nothing replaced the metric. SemDist in 2021 measured semantic distance in an embedding space and correlated better with downstream understanding than WER did.² More recent work has built frameworks for injecting transcript noise and watching which kinds propagate.⁴ This year it was rediscovered again in a clinical setting: Zheng and colleagues, building post-ASR correction for dysarthric speech, report flatly that WER is highly sensitive to domain shift while semantic metrics correlate more closely with downstream task performance.⁶ The field is not short of evidence. It is short of a number that is as easy to report as WER.

WER survived for a good reason, and it is worth saying clearly: for dictation it is still the right metric. If the transcript is what you are shipping, a wrong word on the page is exactly the harm, and every wrong word is roughly as bad as any other. The problem is that almost nobody is shipping transcripts any more. The transcript is now an intermediate representation on the way to an action, and it is being scored as though it were the deliverable.

Anchor Error Rate

Here is what we think should be on the dashboard instead. It is deliberately boring, because a metric that is hard to compute does not get adopted, and WER’s durability is mostly a story about how cheap it is to run.

For a reference utterance, mark its anchors: the spans whose alteration changes what gets done. In practice they fall into four kinds — referents (the people, files, systems and places being pointed at), quantities (numbers, dates, times, doses), conditions (negation, modality, unless, only if), and order (the connectives that fix what happens before what). Everything else in the sentence is scaffolding.

Anchor Error Rate is then the fraction of a reference’s anchors that are not recoverable from the hypothesis. Not identical to — recoverable from. A homophone a competent reader resolves without hesitating is not an error for this purpose, and the 0.4-point measurement says it should not be scored as one.

It is worth being precise about which homophones actually reach you, because most of them do not. A decoder with a language model in the beam resolves them before a transcript exists — send the numbers to the bored does not survive its own search. What survives is the case where both readings parse: proper nouns, quantities, and negation, where can’t and can differ by an unreleased consonant and both continue the sentence perfectly well. So the homophones you actually have to score are not a random sample of the ones a controlled substitution tests, and the 0.4 points should be read as the cost of that operator rather than the cost of every homophone you will ever meet.

Report a second number beside it: fabricated anchors, the count of anchors present in the hypothesis that were never in the reference. This is the number that catches rewriting tools, and it is the one WER is worst at, because it files a confidently invented deadline under “insertion” and weights it the same as a stray um.

The test of a metric is whether it orders known outcomes correctly, and this one does on all three results: near-zero for the homophone, exactly zero for the stripped fillers, high with fabrications for the fluent rewrite. WER orders the same three backwards.

We are not the first to try this, and it would be dishonest to present anchors as arriving out of nowhere. There is a family of metrics that reweight the edit distance rather than replace it — Semantic-WER and the named-entity-weighted variants around it — and they share the founding instinct that tokens are not interchangeable.⁷ Two things separate anchors from that family. The first is that we score recoverability rather than identity: a weighting scheme still counts the homophone as an error and only argues about what it should cost, where the 0.4-point measurement says the honest cost is close to none. The second is coverage. Named entities do not include negation, modality or ordering, and do not send this until legal approves contains no entity whose corruption a weighted edit distance would notice. Conditions and order are exactly where speech carries the most and a transcript carries the least.

The two numbers we run in production

Anchor Error Rate is an offline metric. It needs a reference, and a reference means somebody said the sentence and somebody else marked it up. That works for a benchmark and it does not work for a Tuesday afternoon, because in production nobody says the sentence twice and there is nothing to score against. So the two numbers we watch live do not use a reference at all.

Edit rate is the share of what we produce that the person changes by hand before they accept it — the recipient they swapped, the time they corrected, the clause they retyped. It costs nothing to collect, because the edit is work they were going to do anyway. We are only counting it.

It also carries a bias severe enough that it has to be stated before the number is ever quoted: people can only edit what they can see. If a condition was dropped, there is nothing on the screen to correct, and the edit rate goes down. If a request was rewritten into something fluent and slightly wrong, it looks finished, and the edit rate goes down. Edit rate is a good detector of visible wrongness and a poor detector of missing rightness — which is to say that on its own it fails in the same direction WER does, in a new coordinate system.

Outcome success rate is what closes that gap: whether the thing that got done was the thing that was meant. A little of it is explicit — an undo, a correction, a thumbs-down. Most of it is not. The strongest signal is a request re-issued in different words within a few minutes; after that, an action reverted, a draft that never gets sent, a task reopened by the person who filed it. None of these asks anyone to fill in a survey. Each one is somebody telling you the outcome was wrong by doing something about it.

Neither number is worth much alone. Together they are worth a great deal, because the thing you actually need to know is which quadrant you are in.

The dark cell is the one to be afraid of, and it is the reason this post exists. Low edit rate with low success means nothing on the screen looked wrong, so nothing got corrected, and the outcome was wrong anyway. That is the fluent rewrite: the operator that costs twenty-four points, reads better than what was actually said, and scores best of the three on WER.

Then the loop closes back on the anchors. When someone re-issues a request, the second utterance is an approximate reference for the first, and the place they add specificity — no, the Q3 deck — names the anchor the first attempt lost. An undo marks one that was fabricated. Production disagreement hands you anchor labels without an annotator: noisy ones, biased towards the failures somebody happened to notice, but free and continuous, and they go back into the offline set that Anchor Error Rate is scored on. The offline number tells us what to fix. The two production numbers tell us whether fixing it mattered.

The consequence for how we build has been narrow and specific: nothing in our pipeline is allowed to paraphrase a request. Anchors travel verbatim from the audio through to the action, and any stage that would improve the wording of an instruction has to earn it against outcome success rather than against how the transcript reads.

That is a rule about anchors, not a rule against rewriting, and the difference matters. The Judge-Editor work above rewrites a great deal — it keeps the spans it is confident about and rewrites only the uncertain ones — and it gains 7.66 points of slot F1 doing so, against a 14.51% reduction in WER that is the least interesting number in the paper.⁶ Rewriting is not the failure mode. Rewriting that is free to move an anchor is.

What we have not shown

Anchors have to be annotated, and that is a real cost we are not going to talk around. The mitigation is that you annotate a reference once per benchmark item and then score every system against it forever, which is the same economics as any other labelled evaluation set. “Recoverable” also needs an adjudicator, and a model is the obvious candidate, which introduces an error term of its own that has to be characterised rather than assumed away.

The larger gap is that we have not yet published a correlation between Anchor Error Rate and downstream task accuracy at the scale the results above were measured at. Until we have, this is a well-motivated proposal and not a validated metric, and we would rather say so than imply otherwise. That measurement is the next thing we are building, and we will publish it whichever way it comes out.

In the meantime, the practical version of this post is one sentence. If you are building on top of transcription, the number on your dashboard should be the one that moves when your outcomes move. Ours did not, so we stopped using it.