9. Measuring streaming ASR

What you will learn

• How word and character error rates are computed, why they can exceed 100%, and why every one of them needs the text normalizer that produced it.
• The extra numbers a streaming recognizer needs beyond accuracy: partial stability, first-token delay, final latency and real-time factor, and how to measure them without a microphone.
• How to read published streaming results: which decoding pass, which chunk convention, which test set.
• The book's own measurements of both tracks, in English and Mandarin, on one CPU and one RTX 4090.

Before you start

• Chapters: 3 (latency vocabulary), 5 (chunk conventions), 6 (partial and final results).
• Tools: the ssh-asr environment for the CPU track, ssh-nemo and a GPU for the GPU track (code/README.md).

Word error rate, counted by hand

A recognizer's output is compared with a reference transcript by aligning them word by word with the fewest edits. Each edit is a substitution (S, a wrong word), a deletion (D, a missing word) or an insertion (I, an extra word). The word error rate is

WER = (S + D + I) / N,

where N is the number of words in the reference. jiwer, the Python scorer used here, computes exactly wer = float(S + D + I) / float(H + S + D), H being the correctly recognized words (jiwer process.py, line 247). NIST's sclite, the reference scorer of the field, divides every error type by the number of reference words in the same way.

An example. Reference: "the cat sat on the mat", 6 words. Hypothesis: "the cat sit on mat now".

Reference	the	cat	sat	on	the	mat
Hypothesis	the	cat	sit	on		mat	now
Edit			S		D		I

Three edits over six reference words: WER = 3/6 = 50%. Because insertions have no limit, WER can exceed 100%: reference "hello", hypothesis "oh hello there friend" has three insertions, a WER of 300%.

For a test set, add the edits of all utterances and divide by all reference words. This pooled WER weights each utterance by its length. Some evaluation scripts average the per-utterance WERs instead, which lets a two-word utterance count as much as a fifty-word one. The book always pools, and so should you; check which one a published number used.

Character error rate (CER) is the same computation over characters. It is the standard for Mandarin, which has no spaces between words. Reference 今天天气很好 (6 characters), hypothesis 今天天汽好吗: 气 → 汽 is a substitution, 很 is deleted, 吗 is inserted. CER = 3/6 = 50%.

The normalizer is part of the number

"Mr. Smith's 2 cats." and "mister smiths two cats" are the same speech. Scored as raw strings they share no word. Before scoring, both the reference and the hypothesis go through a text normalizer, and the choice of normalizer can move WER by more than the difference between two models.

Whisper's paper states the goal: "a best-effort attempt to penalize only when a word error is caused by actually mistranscribing a word, and not by formatting or punctuation differences" (arXiv:2212.04356, Appendix C). Its English normalizer, in order (whisper/normalizers/english.py, lines 526-550): lowercase; remove text in brackets; remove fillers (hmm, mm, mhm, mmm, uh, um); expand contractions and titles ("won't" → "will not", "mr" → "mister"); remove symbols except .%$¢€£; write numbers as digits; map British to American spelling; collapse whitespace. The Open ASR Leaderboard uses a normalizer "closely following that of Whisper", and its code is a modified copy with a longer filler list and extra rules for compound words, names and acronyms (open_asr_leaderboard normalizer.py). So "the Whisper normalizer" already means two different things.

The defaults of the scoring tools do nothing of the kind. jiwer's default WER transformation only collapses whitespace: it does not lowercase or remove punctuation. Its CER "by default ... includes space () as a character" (process.py, lines 327-329).

Mandarin has its own choices

For Mandarin, five choices decide what counts as an error: punctuation; full-width against half-width characters (，against ,); traditional against simplified characters; numerals (2019 against 二零一九); and embedded English words (one token, or one per letter). The toolkits make them differently:

• icefall scores AISHELL-1 by joining the words and splitting into characters (ref = list("".join(ref)), icefall/utils.py, lines 738-743). It removes no punctuation, and it labels the result "%WER" although it is a CER.
• WeNet scores with tools/compute-wer.py --char=1, which removes a fixed list of Chinese and ASCII punctuation (but not the ASCII period), removes spaces, makes each Chinese character a token, keeps Latin words whole and uppercases them (compute-wer.py). Its sibling compute-cer.py splits English words into letters instead.

The same hypothesis can therefore score differently under icefall's and WeNet's scripts.

The book's normalizers

Every WER and CER in this book is printed with the name of its normalizer. The main ones are defined in code/hbasr/textnorm.py, each as an ordered list of steps applied to both reference and hypothesis:

Name	Steps
en-raw	collapse whitespace
en-lower	lowercase
en-lower+punct (default for English)	drop language tags, full-width to half-width (NFKC), lowercase, remove punctuation and symbols except apostrophes inside words
zh-raw	collapse whitespace
zh-nospace	remove spaces
zh-nospace+punct	remove punctuation, remove spaces
zh-full (default for Mandarin)	drop language tags, NFKC, traditional to simplified, lowercase, Arabic to Chinese numerals, remove punctuation, remove spaces

Mandarin is scored one token per Chinese character and one per embedded Latin word or number; a space is never a token.

How much does the normalizer matter in practice? It depends on how the model and the references write text. I scored the CPU track's greedy-search outputs under every normalizer (code/ch09_measure/results/sherpa-zipformer-*-greedy_search.json):

Test set (200 utterances)	Normalizer	Errors / reference units	Rate
LibriSpeech test-clean	en-raw, en-lower, en-lower+punct	130 / 4,091 words	3.18% WER under all three
AISHELL-1 test	zh-raw, zh-nospace, zh-nospace+punct	135 / 2,948 characters	4.58% CER
AISHELL-1 test	zh-full	134 / 2,948 characters	4.55% CER
AISHELL-1 test	every character, spaces included	1,759 / 4,574 characters	38.46% CER

The English normalizers change nothing here, because this model was trained on LibriSpeech's own text format, uppercase without punctuation, and the references have the same format. The GPU track's FastConformer writes lowercase: scored raw against the same references its WER on test-clean at [70,13] is 100.4%, almost every word a substitution, and 2.69% after en-lower. Nemotron 3.5 ASR on AISHELL-1 writes punctuation, puts spaces between some characters and appends a language tag: raw, its CER at [56,13] is 41.7%; with zh-full, 12.4%. The last row is the trap: AISHELL-1's transcripts separate words with spaces and the model's output has none, so a scorer that treats spaces as characters counts every space as a deletion, and an honest 4.6% turns into 38.5%. jiwer's CER counts spaces as characters by default, so it falls into exactly this trap.

The numbers a streaming recognizer adds

Accuracy of the final result is half the story. Chapter 3 named the rest; this is how each is measured.

Partial stability. A partial result that changes after the user has read it is distracting. Shangguan et al. defined two rates for this (arXiv:2006.01416, Sec. 2): the unstable partial word ratio (UPWR), "the ratio of total number of unstable words in a test corpus to the total number of words in the final hypotheses", and the unstable partial segment ratio (UPSR), "the ratio between the aggregated number of revised segments and the total number of utterances". A partial word is unstable if a later partial changes it, and so is every word after it in that partial, even if its text is the same. A worked example: the partials "I", "I scream", "ice cream", "ice cream please", final "ice cream please". The third partial revises both words shown before it: 2 unstable words, 1 revised partial. UPWR = 2/3 = 0.67; UPSR = 1/1 = 1.0.

Stability trades against delay. In the same paper, showing partials every 200 ms instead of every 50 ms added 75 ms to the mean partial delay (Sec. 5.1, Figure 1) and cut UPWR by 71.6% and UPSR by 68.8% (Sec. 5.1), at the same WER of 3.6 (Table 4).

First-token delay and final latency. From the start of speech to the first token, and from the end of speech (not the end of the file) to the final result, as chapter 3 defined them. Both need a clock and a known speech start and end. The book's harness gets the start and end from Silero VAD run offline on the original utterance.

Real-time factor. Processing time over audio duration, for one stream and at the concurrency you care about.

Measuring without a microphone

A real microphone delivers 160 ms of audio every 160 ms. A test that sleeps to imitate it takes as long as its audio: 200 utterances of LibriSpeech test-clean are about 25 minutes. The book's harness (code/hbasr/latency.py) instead simulates the pacing: it keeps a virtual clock, says each chunk "arrives" at the moment its audio would have been spoken, and records when the recognizer would have started and finished it, given the measured compute time. Chapter 3 checked that real-time and simulated pacing give the same partials after every chunk and the same final text; only the compute times differ, by as much as a shared CPU varies from run to run.

Simulation is valid only while the recognizer keeps up. If a chunk takes longer than 160 ms, a real system falls behind and every later chunk waits. The harness records that wait as backlog, the time between a chunk's arrival and the start of its processing. Every run below records each utterance's maximum backlog (latency.max_backlog_s in its result file); across all 26 runs the 95th percentile of it was at most 13 ms, a small fraction of one 160 ms chunk, so every recognizer here kept up.

NVIDIA's Riva measures something different under the name "latency": the time from sending a chunk to receiving the matching response, recv_times[i] - send_times[i] (in streaming_recognize_client.cc, lines 331-340, of nvidia-riva/cpp-clients at commit 3911b0b), reported separately for partial (is_final == false) and final responses. That is the server's turnaround per chunk, not the delay of a word. Neither is wrong; they answer different questions.

Reading published results

Three questions get you through most streaming result tables.

Which pass is it? WeNet's U2++ Conformer on AISHELL-1 at chunk 16 reports 5.81% for CTC greedy decoding and 5.05% for attention rescoring (examples/aishell/s0/README.md, lines 18-33). The first is the quality of the partial results; the second, of the final result after the second pass (chapter 6). Quoted without its label, "5.81%" undersells the final result and oversells the partials.

What does "chunk" count? WeNet's chunk 16 is 16 frames of 40 ms, 640 ms. icefall's current Zipformer recipe counts its --chunk-size at 50 Hz, so chunk 16 is 320 ms (egs/librispeech/ASR/RESULTS.md, lines 821-834). NVIDIA's FastConformer card labels each setting by its right context ([70,13] is "1040ms"), while the Nemotron cards label it by its chunk ((right + 1) × 80 ms, so [70,13] would be 1.12 s). Chapter 3 showed these are all correct for what they count.

Which test set and which average? Nemotron 3.5 ASR's "10.38% at 80 ms, 8.84% at 1.12 s" is the average over 19 locales of FLEURS with the language given; for English alone the same model card gives 9.43% and 7.91%, and for Mandarin CER 20.56% and 19.28% (nvidia/nemotron-3.5-asr-streaming-0.6b README, lines 686-713). The Open ASR Leaderboard, the usual place to compare English models, has no streaming or latency track; its speed number, RTFx, is measured on an A100 at batch size 64 where memory allows (smaller otherwise) (arXiv:2510.06961, Secs. 2-3).

Within each family the shape is the same: accuracy improves as the chunk grows, quickly at first and then slowly. icefall's Zipformer on LibriSpeech test-other goes from 7.79% at 320 ms to 7.16% at 640 ms (greedy, chunk-wise decoding); Nemotron's English WER from 9.43% at 80 ms to 7.99% at 560 ms, and only to 7.91% at 1,120 ms.

Test sets and their licences

The book measures on fixed 200-utterance subsets of LibriSpeech test-clean and test-other (English, CC BY 4.0, openslr.org/12) and AISHELL-1 test (Mandarin, Apache 2.0, openslr.org/33), listed in code/data/manifests/. Before you publish numbers on other corpora, read their terms: WenetSpeech is released "for non-commercial purposes" (its official site, although openslr.org/121 lists plain CC BY 4.0); GigaSpeech's download terms allow "only ... non-commercial research and educational purposes" although its Hugging Face metadata says apache-2.0; TED-LIUM 3 is CC BY-NC-ND 3.0.

The book's measurements

Both tracks, streamed with the harness at simulated real-time pacing in 160 ms client chunks, scored with en-lower+punct and zh-full. The CPU track runs sherpa-onnx streaming Zipformers on one thread of an Intel i5-12600K; the GPU track runs NeMo cache-aware FastConformer and Nemotron 3.5 ASR through the book's incremental adapter (chapter 5) on one RTX 4090. Every row comes from a result file in code/ch09_measure/results/ that records its command, package versions, hardware and load.

English, LibriSpeech, 200 utterances per set. Word latency is the time at which a correctly recognized word first appears in a partial result that never changes it again, minus the end of that word in a wav2vec 2.0 forced alignment of the reference; it includes the chunk, the look-ahead, compute and emission delay.

System	Setting	test-clean WER	test-other WER	First token p50 / p90 (clean)	Word latency p50 / p90 (clean)	UPWR (clean)	RTF (clean)
Zipformer, CPU, greedy	320 ms chunk	3.18%	8.08%	631 / 887 ms	582 / 742 ms	0.30	0.075
Zipformer, CPU, beam 4	320 ms chunk	3.10%	7.79%	631 / 887 ms	583 / 737 ms	0.30	0.077
FastConformer transducer, GPU	[70,0]	3.42%	7.91%	532 / 755 ms	419 / 599 ms	0.42	0.122
FastConformer transducer, GPU	[70,1]	3.15%	6.92%	522 / 746 ms	390 / 551 ms	0.39	0.067
FastConformer transducer, GPU	[70,6]	2.69%	5.79%	715 / 939 ms	571 / 851 ms	0.11	0.022
FastConformer transducer, GPU	[70,13]	2.69%	5.42%	716 / 939 ms	751 / 1,251 ms	0.06	0.013
FastConformer CTC, GPU	[70,0]	4.67%	9.56%	627 / 818 ms	480 / 640 ms	0.54	0.118
FastConformer CTC, GPU	[70,13]	3.10%	5.91%	746 / 1,095 ms	810 / 1,310 ms	0.07	0.012

Mandarin, AISHELL-1 test, 200 utterances, scored with zh-full:

System	Setting	CER	First token p50 / p90	UPWR	RTF
Zipformer, CPU, greedy	320 ms chunk	4.55%	599 / 823 ms	0.000	0.079
Zipformer, CPU, beam 4	320 ms chunk	3.53%	600 / 763 ms	0.013	0.082
Nemotron 3.5 ASR, GPU	[56,0]	12.45%	569 / 698 ms	0.000	0.158
Nemotron 3.5 ASR, GPU	[56,13]	12.42%	591 / 723 ms	0.000	0.017

The CPU track's Mandarin model was trained on data sets that include some released only for research or non-commercial use; Appendix C has the details.

UPWR here counts mostly a half-shown last word growing ("fain" becoming "faintly"), not rewrites: in every English row except beam search, every unstable unit was of that kind (chapter 6).

Source: code/ch09_measure/results/summary_table.json (python code/ch09_measure/make_tables.py rebuilds it from the per-run files; the CTC rows at [70,1] and [70,6] and Nemotron at [56,3] and [56,6] are there too).

What the tables say, reading only what was measured:

• A word's latency has two parts, and they trade. Split each word's latency into the wait for the audio of the encoder step that contains its end, and the time after that step until the word appears for good. From the per-word logs of the transducer runs on test-clean (code/ch09_measure/breakdowns.py, results in code/ch09_measure/results/breakdowns.json):

Setting	Chunk	Wait for audio, p50	After the step, p50	Words that appear 160 ms or more after the step	Word latency, p50
[70,0]	80 ms	140 ms	339 ms	91%	419 ms
[70,1]	160 ms	140 ms	171 ms	88%	390 ms
[70,6]	560 ms	380 ms	11 ms	32%	571 ms
[70,13]	1,120 ms	620 ms	11 ms	15%	751 ms

The wait for audio is about half a chunk plus the time to fill the last 160 ms packet, as chunk arithmetic predicts (chapter 3: inside a chunk the look-ahead runs from the chunk length down to zero). What the configuration does not show is the second column. With a long chunk the model, having seen the future, emits each word in the same step. With a short chunk it waits: 88% of words appear 160 ms or more after the step that contained their end. With less look-ahead the words appear later after the step that contained them, so cutting the chunk from 1,120 to 160 ms cut the median word latency by 361 ms, not by 480. - Zero look-ahead was not the fastest. [70,0] and [70,1] wait the same for audio here, because the harness sends 160 ms packets and [70,0] processes its two 80 ms steps together when a packet arrives. The difference is all after the step: [70,0] emits later (339 against 171 ms) and spends more compute per packet (19.3 against 10.5 ms median), for a median word latency of 419 against 390 ms and a higher WER, with both heads (CTC: 480 against 450 ms). No partial result was ever rewritten in either; the words come out later, not differently. - The end of the file helps the long chunk. At [70,13], 259 of 3,997 words (6.5%) became final only when the end of the file flushed the encoder, so their latency was set by the file's trailing silence rather than by the next chunk. In a live stream, where audio keeps coming, those words would wait for the next chunk instead. - Accuracy saturates before latency does. On test-clean, [70,6] and [70,13] have the same WER, 2.69%, 180 ms apart in median word latency. - RTF is not latency. The GPU track's transducer RTF ranges from 0.013 to 0.122 across settings, and the slowest by RTF, [70,0], is among the fastest in word latency. At [70,0] the recognizer runs 12.5 steps per second and spends a median 19.3 ms of compute on each 160 ms packet. - The CPU track is close behind. The CPU Zipformer's 582 ms median word latency at 3.18% WER is just behind the FastConformer's [70,6] on both (571 ms, 2.69%), on one CPU thread against a GPU. - In Mandarin, the GPU track is much weaker. Nemotron 3.5 ASR's CER on AISHELL-1 was 12.4 to 14.0% across settings against 3.5 to 4.6% for the CPU Zipformer; its model card places Mandarin in a broader-coverage tier, and in 5 to 12 of the 200 utterances per setting it emitted runs of a Thai character, in the book's adapter; NeMo's own streaming loop produced the same final text on the one utterance of the first ten where the adapter showed it.

Concurrency, many streams at once, is chapter 15's subject.

Common misconceptions

"WER is a percentage of words, so it is at most 100%." It is edits per reference word; insertions make it unbounded.

"WER is WER." Without the normalizer, a WER is not reproducible. Whisper's normalizer, the Open ASR Leaderboard's and jiwer's default give different numbers for the same output.

"A 5.81% streaming CER on AISHELL-1." For WeNet U2++ that is the first, CTC pass; its final result is 5.05%.

Try it

CPU track

1. bash code/ch09_measure/run_cpu_all.sh runs the CPU-track Zipformers on the three test subsets with greedy and modified beam search. Each run writes per-utterance logs under $BOOK_DATA/runs/ and a summary JSON.
2. python code/ch09_measure/make_tables.py rebuilds the summary table from those files.
3. hbasr.evaluate.score(<run directory>/utts.jsonl, "en") rescores a run under every normalizer at once; the result's accuracy block holds one entry per normalizer. Try it on your own model's output.

GPU track

code/ch09_measure/run_gpu_all.sh runs the NeMo models at [70,0], [70,1], [70,6] and [70,13] on one GPU; its header gives the exact command line. Compare the WER with the published card values above, and the word latency across settings. Do not compare the final latency after end of speech across settings: at the end of a file the recognizer flushes without waiting for look-ahead, so on test-clean its median is 283 to 286 ms for every setting (334 to 345 ms on test-other), the files' trailing silence plus the flush.

English and Mandarin

Score the Mandarin run with zh-raw and then zh-full. Find the utterances whose CER changes, and see which of the five Mandarin choices caused it.

Key terms

• substitution, deletion, insertion (S, D, I): the three edit types of an alignment.
• word error rate (WER): (S + D + I) / reference words; can exceed 100%.
• character error rate (CER): the same over characters; the standard for Mandarin.
• pooled WER: total edits over total reference words for a test set.
• text normalizer: the transformation applied to reference and hypothesis before scoring.
• UPWR, UPSR: unstable partial word ratio and unstable partial segment ratio.
• simulated pacing: replaying audio against a virtual clock instead of sleeping.
• backlog: time between a chunk's arrival and the start of its processing.

Sources

• jiwer v4.0.0 (commit 19e88ea): src/jiwer/process.py, src/jiwer/transformations.py: https://github.com/jitsi/jiwer/tree/19e88eae1815e7ca40db9b45b14af5f1f9d8b6ce
• usnistgov/SCTK (commit 9688a26): doc/sclite.htm: https://github.com/usnistgov/SCTK/tree/9688a26882a688132a5e414cadcb4c19b6fffaba
• Radford et al., "Robust Speech Recognition via Large-Scale Weak Supervision", arXiv:2212.04356; openai/whisper v20250625 (commit 31243ba), whisper/normalizers/english.py.
• huggingface/open_asr_leaderboard (commit 2aaa07a): normalizer/normalizer.py; "Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation", arXiv:2510.06961.
• icefall (commit 9c96676): icefall/utils.py, egs/librispeech/ASR/RESULTS.md, egs/aishell/ASR/RESULTS.md: https://github.com/k2-fsa/icefall/tree/9c966762e058bd7ba79d9da71e42fc06b64b703e
• WeNet (commit be6ad97): tools/compute-wer.py, tools/compute-cer.py, examples/aishell/s0/README.md: https://github.com/wenet-e2e/wenet/tree/be6ad97fdb512589df113042f7ff3a29d392fb38
• Shangguan et al., "Analyzing the Quality and Stability of a Streaming End-to-End On-Device Speech Recognizer", arXiv:2006.01416.
• nvidia-riva/cpp-clients (commit 3911b0b): riva/clients/asr/streaming_recognize_client.cc; Riva 2.19.0 ASR performance documentation.
• NVIDIA model cards: nvidia/stt_en_fastconformer_hybrid_large_streaming_multi (revision ae98143), nvidia/nemotron-3.5-asr-streaming-0.6b (revision ea30d66).
• openslr.org resources 12 (LibriSpeech), 33 (AISHELL-1), 121 (WenetSpeech); the WenetSpeech official site; the GigaSpeech access terms on Hugging Face.