Skip to content

6. Streaming decoding: partials, finals, fusion, biasing

What you will learn

  • How a streaming recognizer turns encoder output into text chunk by chunk, and what partial and final results are.
  • How greedy and beam search work for a transducer, and what the beam actually buys.
  • Why transducers emit late, and how FastEmit and the delay penalty pull emission earlier, at what cost in accuracy.
  • How an external language model, a hotword list and "two-pass" decoding change the result, without retraining.

Before you start

  • Chapters: 4 (the transducer lattice and its alignments) and 5 (streaming encoders and chunks).
  • Tools: sherpa-onnx with the English and Mandarin streaming Zipformer models (CPU track) and NeMo's cache-aware FastConformer (GPU track). See Try it.

Partial and final results

A streaming recognizer gives two kinds of output. A partial result is its current best guess for the utterance so far; it is updated after every chunk and may change. A final result is committed: once the recognizer decides the utterance (or a segment of it) has ended, it outputs the final text and starts a new segment. Chapter 7 is about how it decides that; this chapter is about how it produces the text.

Users see partials as live captions, and partials that keep changing are distracting. A voice agent mostly acts on finals. Both matter, and they can be improved separately.

Greedy search for a transducer

The simplest decoder walks the lattice of chapter 4 left to right. At encoder frame t, it runs the joiner on the current encoder vector and the prediction network's state, and takes the most probable symbol. If it is a label, it emits it, updates the prediction network, and asks again at the same frame; if it is ∅, it moves to the next frame. Because the decoder only ever looks at the current frame, it streams exactly as fast as the encoder.

Two details matter in practice:

Beam search, and what it buys

Beam search keeps several hypotheses instead of one. The variant you will use is icefall's modified_beam_search: "Beam search in batch mode with --max-sym-per-frame=1 being hardcoded" (icefall egs/librispeech/ASR/pruned_transducer_stateless2/beam_search.py line 1207, commit 9c96676). At each frame it extends every hypothesis by every symbol (∅ keeps the hypothesis as it is), keeps the beam best over all of them (default 4, line 1202), and merges hypotheses that have reached the same text by adding their probabilities (lines 846-849). Its closest published description is the one-symbol-per-frame beam search of arXiv:2211.00484 (Secs. 2.3 and 2.5). In sherpa-onnx it is decoding_method="modified_beam_search" with max_active_paths=4 (online-recognizer.h).

What does the beam buy? On LibriSpeech (non-streaming encoder, no language model), the same paper measured (Table 1, test-clean/test-other WER):

decoding WER
greedy, one symbol per frame 2.72 / 6.33
beam, one symbol per frame 2.67 / 6.21
greedy, unlimited symbols 2.68 / 6.28
beam, unlimited symbols 2.65 / 6.18

and, batched on a V100, a real-time factor of 0.0009 for greedy against 0.0068 for beam with one symbol per frame (Table 5). 0.05 to 0.12 WER points for 7.6 times the compute. I found no published streaming version of this comparison; the CPU track below measures one.

The beam matters more once you add something outside the model: a language model or a hotword list. Both work by changing scores inside the beam, so with greedy search they have little to work on. sherpa-onnx enforces this: it rejects a hotwords file unless modified_beam_search is used (online-recognizer.cc line 149).

Why transducers emit late

Recall the three alignments of "ab" in chapter 4: they differ only in when each label is emitted, and the loss does not care. A streaming encoder with some look-ahead gives the model a reason to wait: the later it emits, the more future audio it has seen, and the more confident it is. Models learn this. A study of transducers with unidirectional LSTM encoders (Mahadeokar et al., "Alignment Restricted Streaming Recurrent Neural Network Transducer") found they "tend to wait for longer spans of input audio, before streaming already decoded ASR tokens", with an average emission delay of 263 ms (arXiv:2011.03072, abstract and Sec. 5).

That delay is invisible in WER and fully visible to users: captions lag, and the final result comes later. Two training changes push emission earlier.

FastEmit (arXiv:2010.11148) adds to the loss a bonus for the probability mass that flows through label emissions, weighted by one hyper-parameter λ (Eqs. 9-10). It measures latency as the time from the end of speech to the emission of the last token of the final result, which can be negative if the model emits before the word ends. On LibriSpeech with a streaming ContextNet, median latency went from 50 to −120 ms and the 90th percentile from 210 to 30 ms, while WER improved from 4.4/8.9 to 3.1/7.5% (Table 1). On Google's Voice Search data, WER instead rose by 0.2 points (Table 3). The effect on accuracy depends on the data. Partial-result latency fell in every row, but the endpointer's latency did not always (Conformer transducer: EP50 260 to 290 ms, EP90 590 to 660 ms, Table 3).

The delay penalty (arXiv:2211.00490), used in icefall, adds λ · ((T − 1)/2 − t) to the log-probability of every label emission at frame t before the forward recursion (Eq. 19): early emissions are rewarded, late ones penalized. On a streaming Conformer with 640 ms chunks (LibriSpeech test-clean):

λ WER MAD MED
0 3.40 373 ms 485 ms
0.003 3.70 102 ms 176 ms
0.006 3.74 −3 ms 62 ms
0.01 4.67 −93 ms −37 ms

(Table 1. MAD is the mean delay over all correctly recognized words; MED, the mean end delay, averages the delay of each utterance's last word (Eqs. 20-21). Both delays "are the total delays including the latency introduced by the chunk-wise decoding, which equals half of the chunk length".) The paper finds "similar delay-accuracy trade-offs" to FastEmit.

The delay penalty by hand

Apply the penalty to the lattice of chapter 4, two frames numbered from 0 (T = 2) and target "ab". An alignment's delay score is d = Σ_u ((T − 1)/2 − t_u), where t_u is the frame at which label u is emitted (Eq. 18). Adding λ(T − 1)/2 − λt to each emission's log-probability multiplies the alignment's probability by e^{λd}:

alignment emitted at d probability × e^{0.1·d} (λ = 0.1)
a b ∅ ∅ 0, 0 +1 0.1728 0.1910
a ∅ b ∅ 0, 1 0 0.1890 0.1890
∅ a b ∅ 1, 1 −1 0.1323 0.1197

The earliest alignment's share of the total rises from 0.1728 / 0.4941 = 35.0% to 0.1910 / 0.4997 = 38.2%. Training on this modified sum moves the model toward the early alignment.

Emission delay against accuracy for the delay-penalized transducer (arXiv:2211.00490, Table 1): as λ grows from 0 to 0.01, the mean delay falls from 373 ms to −93 ms and WER rises from 3.40% to 4.67%.

Adding an external language model

A recognizer trained on a few thousand hours of speech has seen far less text than a language model trained on text alone. Shallow fusion adds the language model's score during decoding:

Ŷ = argmax_Y [ log P(Y | X) + λ_T log P_LM(Y) ]

(arXiv:2011.01991, Eq. 11). The transducer, however, already contains an internal language model in its prediction network, which double-counts with the external one. Internal language model estimation (ILME) subtracts an estimate of it, computed from the prediction network alone (Eqs. 18 and 26). With 30,000-hour transducer and attention models, ILME gave "up to 15.5% and 6.8% relative word error rate (WER) reductions from Shallow Fusion for cross- and intra-domain evaluations, respectively": the 15.5% on LibriSpeech, outside the models' training domain, the 6.8% on Microsoft production test sets inside it.

In sherpa-onnx, an LM is used through modified_beam_search with an lm_scale (default 0.1, sherpa-onnx/python/sherpa_onnx/online_recognizer.py lines 116-125). The LM is evaluated for every hypothesis in the beam as decoding proceeds, so check the real-time factor when you add one.

Hotwords and contextual biasing

Names, product terms and contact names are exactly the words a general model gets wrong and a user cares about. Contextual biasing boosts a given list of phrases during decoding.

  • sherpa-onnx / icefall hotwords build an Aho-Corasick graph, "mainly a Trie with a fail arc for each node", over the hotword tokens, and add a bonus for every matched token: "The bonus score for each token (note: NOT for each word/phrase, it means longer word/phrase will have larger bonus score, they have to be matched though)" (icefall icefall/context_graph.py lines 81-108). The default bonus is 1.5 per token (hotwords_score, online-recognizer.h).
  • NeMo word boosting (GPU-PB) is the same idea on the GPU: "GPU-PB is applied only at the decoding step in shallow fusion mode. You do not need to retrain the ASR model", with a recommended context_score of 1.0, and per-stream boosting "including cache-aware streaming models" (docs/source/asr/asr_customization/word_boosting.rst lines 9-45 and 151-157, NeMo v3.0.0).
  • Learned biasing builds the list into the model. CLAS "outperforms the baseline method by as much as 68% relative WER" against shallow fusion with contextual n-grams (arXiv:1808.02480, abstract). It requires training, so this book uses the decoding-time methods.

Biasing has a cost: a large bonus makes the recognizer hear hotwords that were not said. Tune it on audio that contains the hotwords and audio that does not.

A hotword graph for two phrases: a trie over their tokens with a fail arc from every node, and the per-token bonus collected along a match.

Two-pass decoding improves the final, not the partials

A two-pass system runs a streaming first pass for partials and, when the segment ends, rescores the first pass's best hypotheses with a stronger model that sees the whole segment. Google's two-pass model rescored a streaming transducer with an attention decoder and "achieves a 17%-22% relative reduction in WER compared to RNN-T alone", adding "less than 200ms" to finalization (arXiv:1908.10992). WeNet's U2 does the same with a CTC first pass: "The CTC hypotheses are then rescored by the attention decoder to get the final result" (arXiv:2012.05481, abstract). On AISHELL-1, U2++ scores 5.05% CER for the rescored streaming final against 5.81% for its first pass alone (WeNet results; see chapter 9).

Read that carefully: the rescoring changes only the final result. The partials the user watched were the first pass's. A later paper on partial rewriting makes the point directly: in multi-stage systems "streaming results do not benefit from the quality improvements" (arXiv:2312.09463, abstract).

Common misconceptions

"Beam search makes a streaming recognizer much more accurate." On its own, beam search improved WER by 0.05 to 0.12 points in the published comparison, at 7.6 times the compute. Its real value is as the place where a language model or a hotword list can act.

"A transducer emits each word as soon as it has heard it." Trained without a latency term, it learns to wait, by hundreds of milliseconds. FastEmit or the delay penalty is what makes early emission part of the objective.

"Two-pass rescoring gives better live captions." It gives a better final result. The partials are still the first pass's. If the captions are what users judge, improve the first pass.

Try it

The scripts are in code/ch06_decoding/ and code/ch09_measure/ (environment ssh-asr, see code/README.md). My numbers below come from their result files; every run streams 200 test utterances in 160 ms chunks with simulated real-time pacing on one CPU thread.

CPU track

  1. Greedy against beam. bash code/ch09_measure/run_cpu_all.sh decodes the English and Mandarin test sets with sherpa-onnx's streaming Zipformers, once with greedy_search and once with modified_beam_search (beam 4):
Test set Greedy Beam 4 RTF greedy RTF beam
LibriSpeech test-clean, WER 3.18% (130 / 4,091) 3.10% (127) 0.075 0.077
LibriSpeech test-other, WER 8.08% (279 / 3,452) 7.79% (269) 0.075 0.078
AISHELL-1 test, CER 4.55% (134 / 2,948) 3.53% (104) 0.079 0.082

Beam search helped little in English and cut the Mandarin error rate by almost a quarter, with a real-time factor within 5% of greedy's (2.5% to 4.4% higher). The first-token delay stayed near 0.63 s in English (p50; 0.66 s for greedy on test-other) and 0.60 s in Mandarin. 2. How partials change. The same runs record every partial result. Scored as words, the English greedy partials have an unstable partial word ratio (UPWR: the share of final words that some partial showed differently; chapter 9) of 0.30 on test-clean; the Mandarin greedy partials, scored as characters, have 0.00. Look at the logs before you conclude that English partials are unstable: a greedy transducer never takes back a token it has emitted, so almost all of the English "revisions" are the last word growing. In one test-other log the partial reads "I REALLY WAS VERY MUCH A" at 2.42 s and "... AFRAID" at 2.74 s (code/ch06_decoding/results/partials_finals.json). Across the 200 test-clean greedy logs, none of the 3,903 changes between consecutive partials rewrote the text: each one appended a word or, 1,120 times, extended the last one. In Mandarin the greedy partials never rewrite a finished character either, but a character is not always one token: the Mandarin model's vocabulary spells rarer characters as two or three byte tokens (圳 is <0xE5><0x9C><0xB3>), and until the last byte arrives the partial shows a replacement character (�). 62 partials in 23 of the 200 utterances showed one. The scorer strips it as a symbol, which is why the Mandarin UPWR is 0.00; a display should hold such a character back until it is complete.

The same byte tokens expose a trade-off at the end of a stream. The exported model only processes whole chunks, so to get the last words out, the client appends silence before closing the stream; sherpa-onnx's examples append 0.66 s, and so does the book's harness. With 0.66 s, 2 of the 200 Mandarin greedy results ended two bytes into a three-byte character: "…的融资渠道" came out as "…的融资��". With 1.5 s the characters completed, but the extra silence let the model keep talking: final results gained words nobody said ("追踪" became "追踪人员", "评估" became "评估报告", "the same result" became "the same result and"). Insertions rose from 21 to 27 on LibriSpeech test-clean (greedy) and from 0 to 6 on AISHELL-1 (beam search) (code/ch09_measure/results/tail_padding_1.5s/). The book keeps the upstream 0.66 s. In a live system the endpointer, not the client, ends the utterance, and the same question becomes how much trailing silence the recognizer sees before the endpoint fires (chapter 7). Beam search adds real revisions, when the best beam switches: Mandarin UPWR rises from 0.000 to 0.013. To show users stable text, hide the last word until a space or the next word has arrived. 3. Hotwords. python code/ch06_decoding/hotwords.py --manifest librispeech-test-other-200 (and aishell1-test-200) builds a hotword list from the reference words the beam-search baseline got wrong (at least 6 letters in English, 2 characters in Mandarin) and decodes again with that list at the default score of 1.5. Because the list comes from the test set itself, the gain on its words is an upper bound; the number to watch is what the list does to everything else:

Test set Listed words Recognized occurrences Error rate, utterances with a listed word Error rate, the other utterances
LibriSpeech test-other, no list 86 19 of 114 11.84% 3.80%
LibriSpeech test-other, with list 86 61 of 114 9.22% 3.97%
AISHELL-1 test, no list 48 4 of 56 9.22% 1.32%
AISHELL-1 test, with list 48 8 of 56 8.01% 1.32%

In English the list recovered 42 more occurrences of its words and cost 3 extra errors on the 125 utterances that contain none of them. In Mandarin the same score recovered only 4 more. It is not because Mandarin hotwords are shorter in tokens: under the model's bpe.model the listed Mandarin words average 3.85 tokens and the English ones 4.29. Raise --score for Mandarin and watch the last column of errors.

GPU track

Run NeMo's cache-aware FastConformer with greedy_batch and with beam search on the GPU, and with word boosting for the same hotword list. The book measured the greedy baseline in chapter 9 (5.42% WER on its LibriSpeech test-other subset at [70,13], through code/ch09_measure/eval_nemo.py); compare your beam-search and boosted runs with it on the same subset, and check what each costs in RTF and word latency.

English and Mandarin

Hotwords recovered far fewer words in Mandarin at the same score, although the listed words have about as many tokens in both languages (a rare character takes up to three byte tokens). Compare the score you need for the same recall in each language.

Key terms

  • partial result: the current, revisable best guess for the utterance so far.
  • final result: the committed text of a finished segment.
  • greedy search: take the most probable symbol at each step.
  • modified beam search: icefall's transducer beam search with one label per frame and merged equal hypotheses.
  • emission delay: how late a model emits a label relative to when it was spoken.
  • FastEmit: a training regularizer that rewards emitting labels early.
  • delay penalty: a training term that rewards early and penalizes late label emissions in the lattice.
  • shallow fusion: adding an external language model's score during decoding.
  • internal language model estimation (ILME): subtracting the model's own built-in language model when fusing an external one.
  • contextual biasing / hotwords: boosting a given list of phrases during decoding.
  • two-pass decoding: a streaming first pass for partials, rescored at the end of a segment for the final.

Sources

  • Fast and parallel decoding for transducer, arXiv:2211.00484: https://arxiv.org/abs/2211.00484
  • FastEmit, arXiv:2010.11148: https://arxiv.org/abs/2010.11148
  • Delay-penalized transducer, arXiv:2211.00490: https://arxiv.org/abs/2211.00490
  • Alignment Restricted Streaming Recurrent Neural Network Transducer, arXiv:2011.03072: https://arxiv.org/abs/2011.03072
  • Internal language model estimation, arXiv:2011.01991: https://arxiv.org/abs/2011.01991
  • CLAS, arXiv:1808.02480: https://arxiv.org/abs/1808.02480
  • Two-pass end-to-end speech recognition, arXiv:1908.10992: https://arxiv.org/abs/1908.10992
  • U2, arXiv:2012.05481: https://arxiv.org/abs/2012.05481
  • Partial rewriting for multi-stage ASR, arXiv:2312.09463: https://arxiv.org/abs/2312.09463
  • icefall, commit 9c96676 (pruned_transducer_stateless2/beam_search.py, icefall/context_graph.py): https://github.com/k2-fsa/icefall/tree/9c966762e058bd7ba79d9da71e42fc06b64b703e
  • sherpa-onnx v2.0.1 (sherpa-onnx/csrc/online-recognizer.*): https://github.com/k2-fsa/sherpa-onnx/tree/af2d8d81dbfa61c4cea4c568da78f3982edca005
  • NeMo v3.0.0 (rnnt_decoding.py, word_boosting.rst): https://github.com/NVIDIA/NeMo/tree/fd6a877539710e2b98f28c43272ff81312f83417