5. Streaming encoders: masks, look-ahead, caches

What you will learn

• What exactly an encoder has to give up to stream, and why "make the attention causal" is not enough.
• How chunked attention works, how to read a setting such as NeMo's att_context_size: [70, 13], and how many milliseconds of look-ahead it buys.
• Why look-ahead grows with depth in one design and not in the other, and what a streaming encoder keeps in its cache between chunks, so that it never recomputes the past.
• How accuracy changes with look-ahead, from published numbers and from your own run.

Before you start

• Chapters: 2 (feature frames every 10 ms, subsampling), 3 (look-ahead and algorithmic latency), 4 (CTC and transducer outputs, which stream as soon as the encoder does).
• Tools: the CPU track uses sherpa-onnx with an icefall streaming Zipformer; the GPU track uses NeMo with a cache-aware FastConformer on the RTX 4090. See Try it.

What an encoder sees

An encoder turns feature frames into a shorter sequence of vectors. Most modern speech encoders do it in two stages. A subsampling front end of convolutions shrinks the sequence: FastConformer uses "8x downsampling at the start of the encoder" (arXiv:2305.05084, Sec. 2.1), so one encoder step covers 8 × 10 ms = 80 ms. Then a stack of layers, each with self-attention and convolution, lets every step look at the others.

Offline, every step can see every other step: the first word's vector is computed knowing how the sentence ends. That is part of why offline models are more accurate. A streaming encoder must produce its output for the current audio before the future exists. There are three places where an encoder can peek into the future, and a streaming design has to control all three:

1. Self-attention, which by default attends over the whole sequence.
2. Convolutions in the layers, which look at neighbours on both sides unless they are causal.
3. The subsampling front end, whose convolutions also look a few frames ahead.

Making only the attention causal leaves the other two. The icefall streaming Zipformer is a concrete case. Its front end needs 13 feature frames of future audio for every chunk: the subsampling computes "(T - 7) // 2" output frames and "the ConvNeXt module needs (7 - 1) // 2 = 3 frames of right padding after subsampling" (egs/librispeech/ASR/zipformer/streaming_decode.py lines 472-474, icefall commit 9c96676), hence self.pad_length = 7 + 2 * 3 (decode_stream.py line 79). That is 130 ms of look-ahead that no attention mask shows.

Two ways to limit attention

Attention is limited by a mask: a matrix that says which steps may attend to which. NeMo's cache-aware FastConformer (arXiv:2312.17279) implements two main styles (a third, chunked_limited_with_rc, is a variant of the second), and the difference between them is the most important idea in this chapter.

Regular look-ahead is a band: each step attends to a fixed number of steps on its left and a fixed number on its right (triu/tril with those offsets, nemo/collections/asr/modules/conformer_encoder.py lines 818-822 at NeMo v3.0.0). It sounds natural, and it has a catch. Layer 2 attends two steps to the right of layer 1's outputs, which themselves attended two steps to the right of the input. The look-ahead adds up layer by layer. NeMo's documentation gives the arithmetic: "for a model of 17 layers with 4x downsampling and 10ms window shift, then even 2 right context in each layer means 17210*4=1360ms look-ahead" (NeMo v2.7.3 docs, docs/source/asr/models.rst line 207; the same example is in the paper, Sec. 3.1). On an 8x model the same setting would be 17 × 2 × 80 ms = 2,720 ms.

Chunk-aware look-ahead splits the steps into chunks. Every step attends to all steps in its own chunk and to a fixed number of earlier chunks, and to nothing after its chunk. The code, simplified from conformer_encoder.py lines 823-842 (which also handle unlimited left context), is short enough to read:

chunk_size = att_context_size[1] + 1
left_chunks_num = att_context_size[0] // chunk_size
chunk_idx = torch.div(chunk_idx, chunk_size, rounding_mode="trunc")
diff_chunks = chunk_idx.unsqueeze(1) - chunk_idx.unsqueeze(0)
mask = (diff_chunks <= left_chunks_num) & (diff_chunks >= 0)

Because no step ever sees past its chunk boundary, depth adds nothing: "In contrast with the effective look-ahead growing with depth in regular look-ahead, there is no such dependency in chunk-aware look-ahead" (arXiv:2312.17279, Sec. 3.1). Inside a chunk the look-ahead varies: "tokens at the first position of each chunk would see all the next 19 tokens while the last token would see zero future tokens" for a chunk of 20 (NeMo v2.7.3 models.rst lines 209-213). "That is why we recommend to use this approach for streaming" (same place).

Reading att_context_size

NeMo writes the setting as att_context_size: [left, right]. The shipped streaming configuration (examples/asr/conf/fastconformer/cache_aware_streaming/fastconformer_transducer_bpe_streaming.yaml, NeMo v3.0.0) uses n_layers: 17, subsampling_factor: 8, att_context_size: [70, 13], att_context_style: chunked_limited and causal convolutions (lines 86-128). Its own comment gives the formula: "look-ahead(secs) = att_context_size[1]subsampling_factorwindow_stride, example: 1380.01=1.04s" (line 106).

Work it through for [70, 13]:

• chunk size = right + 1 = 14 encoder steps = 14 × 80 ms = 1,120 ms of audio;
• the first step of a chunk waits for the other 13 steps: 1,040 ms of look-ahead, the number in the config comment; the last step waits for none;
• left context = 70 steps, 70 // 14 = 5 earlier chunks, 5.6 s of history.

And for [70, 0]: a chunk of one step, 80 ms, with no look-ahead at all.

The look-ahead varies from 1,040 ms to 0 inside a chunk, so a single "latency" number for a chunk-aware model has to say which convention it uses. Papers usually report the average. Emformer defined it as "the look-ahead context latency plus center block latency discounted by 0.5" (arXiv:2010.10759, Sec. 3.2.1), the encoder-induced latency (EIL). A chunk-aware model has no separate look-ahead block, so the convention must be stated again: the cache-aware paper reports half the worst-case look-ahead (arXiv:2312.17279, Sec. 4.2), which for [70, 13] gives 520 ms, while counting the whole 1,120 ms chunk as Emformer's center block would give 560 ms. Neither is a number NeMo prints. WeNet's two papers show how easily this confuses: the same setting of 16 frames per chunk is called "640ms latency" in U2's abstract, which quotes the chunk length, and "320ms" in U2's own comparison table and in U2++, which says "the model latency of chunk=16 mode is ranged from 0 to 640ms, so the average latency is 320ms" (arXiv:2106.05642, Sec. 4.1). In this book a chunk-aware look-ahead is quoted as the chunk-start (worst-case) value unless it says "average".

All of these are algorithmic latencies, computed as if the network took no time: "Latencies are based on the assumption that the forward time of the network is zero" (NeMo v2.7.3 models.rst). Chapter 9 measures the real thing.

Never recompute the past: caches

A chunked encoder could, in principle, recompute everything from the start of the utterance for each new chunk. That is what buffered streaming does with an offline model: it runs the model on overlapping windows. It works, "however, it causes significant amount of duplication in computation due to the overlapping chunks" (NeMo v2.7.3 models.rst line 182), and the model sees context at inference that it never saw in training.

A cache-aware encoder keeps, between chunks, exactly the intermediate values the next chunk will need, and nothing else. In NeMo's FastConformer that is three tensors per stream (get_initial_cache_state, conformer_encoder.py lines 1087-1125):

• cache_last_channel, shape (layers, batch, 70, d_model): each layer's activations for the last 70 steps, which the attention of the next chunk reads as its left context;
• cache_last_time, shape (layers, batch, d_model, 8): each layer's last 8 inputs to its causal convolution, kernel size 9 minus one;
• cache_last_channel_len: how much of the attention cache is filled.

The paper's Sec. 3.3 gives the same sizes. Because the model was trained with the same masks, "the prediction of the network would be exactly the same as when the entire audio is processed in a single step" (arXiv:2312.17279, Sec. 3.3), and the method is "designed in a way to eliminate the accuracy disparity between the train and inference time" (abstract).

The icefall streaming Zipformer does the same with six cached tensors per layer: "cached_key, cached_nonlin_attn, cached_val1, cached_val2, cached_conv1, cached_conv2"; the four attention caches are sized by the left context and the two convolution caches by half the convolution kernel (egs/librispeech/ASR/zipformer/zipformer.py lines 443-444 and 492-537).

The caches are why a streaming server can hold thousands of streams: each stream is a small block of state, and one batched forward pass advances many streams by one chunk (chapter 15).

The encoders you will meet

• Conformer combines self-attention and convolution in each layer. The original, non-streaming model reached "2.1%/4.3% without using a language model and 1.9%/3.9% with an external language model" on LibriSpeech test/test-other (Interspeech 2020, arXiv:2005.08100).
• FastConformer is a Conformer with "8x downsampling at the start of the encoder", depthwise separable convolutions in the subsampling, fewer channels and a convolution kernel of 9; it is "2.8× faster than the original Conformer" (arXiv:2305.05084). Its cache-aware streaming form is NeMo's main streaming model and this book's GPU exemplar.
• Emformer (arXiv:2010.10759) is an earlier memory-based streaming transformer from Meta. Its headline numbers (2.50/5.62% at 960 ms EIL) are for hybrid systems, not transducers (Tables 2-4); its transducer reached 2.78/6.92% at 640 ms.
• Zipformer (arXiv:2310.11230) runs its middle layers at lower frame rates: after a front end that halves 100 Hz features to 50 Hz, its stacks run "at frame rates of 50Hz, 25Hz, 12.5Hz, 6.25Hz, 12.5Hz, and 25Hz", and the output is 25 Hz (40 ms per step) (Sec. 3.1). It is the k2/icefall model family and this book's CPU exemplar, through sherpa-onnx.

Reading icefall's chunk settings

icefall counts chunks at the 50 Hz front-end rate, not at the 25 Hz output rate. Training samples --chunk-size from "16,32,64,-1", "Chunk sizes (at 50Hz frame rate)", with --left-context-frames from "64,128,256,-1" (egs/librispeech/ASR/zipformer/train.py lines 269-290); decoding uses one value of each. So chunk 16 is 16 × 20 ms = 320 ms of audio, plus 130 ms of feature look-ahead and the last window's 7.5 ms: the first chunk can run after about 458 ms of audio has arrived (a derivation from the code above). A left context of 128 frames is 2.56 s.

Training one model for several latencies

A model trained with one chunk size does best at that chunk size. Dynamic chunk training, introduced with WeNet's U2 (arXiv:2012.05481), samples a different chunk size for each batch: "a uniform distribution from 1 to max utterance length, namely the attention varies from left context attention to full context attention" (Sec. 3.2.2). The same model then runs at any chunk size, including offline. icefall's list of chunk sizes above is the same idea. NeMo's cache-aware models can be trained with several look-aheads; trained on NVIDIA's ASRSET 3.0, a collection of public data sets, one model trained for three look-aheads scored, with its transducer head, 6.2/5.5/5.2% WER on LibriSpeech test-other at 40/240/520 ms against 6.4/5.9/5.4% for three separately trained models (arXiv:2312.17279, Table 4).

What look-ahead buys

The cache-aware paper reports LibriSpeech test-other WER for a 114M-parameter FastConformer (arXiv:2312.17279, Tables 1-2):

setting	algorithmic latency (EIL)	CTC WER	transducer WER
offline	whole utterance	5.7	5.0
buffered streaming of an offline model	1,500 / 2,000 ms	8.0	11.3
zero look-ahead	0 ms	10.6	9.5
regular look-ahead	1,360 ms	7.9	7.1
chunk-aware look-ahead	1,360 ms	7.1	6.3

And for the hybrid transducer at increasing chunk-aware look-ahead: 9.5, 9.0, 7.8, 7.5, 7.3 and 6.3% at 0, 40, 240, 520, 680 and 1,360 ms (Table 2). Three readings:

• The first 240 ms of look-ahead buys most of the gain (9.5 to 7.8%); the next 1,120 ms buys 1.5 points more.
• At the same latency, chunk-aware beats regular look-ahead (6.3 against 7.1%).
• Buffered streaming of an offline model is worse than a model trained to stream, at more latency.

These latencies count only the encoder's own waiting, from the paper's definition: "All the average latencies in this paper are referring to algorithmic latency induced by the encoder (EIL)" (Sec. 4).

The GPU track's FastConformer was trained for four look-aheads, so one model can be measured at all of them. I streamed a 200-utterance subset of LibriSpeech test-other through it on one RTX 4090, in 160 ms chunks with simulated real-time pacing, through the book's incremental adapter, which reproduces NeMo's own cache-aware streaming output (identical final text on 20 of 20 utterances at each setting, code/ch05_streaming_gpu/results/check_adapter_fastconformer-en-multi.json):

att_context_size	Worst-case look-ahead	Transducer WER, measured	Transducer WER, model card	CTC WER, measured	CTC WER, model card
[70,0]	0 ms	7.91%	7.0%	9.56%	8.4%
[70,1]	80 ms	6.92%	6.4%	8.00%	7.8%
[70,6]	480 ms	5.79%	5.7%	6.87%	6.7%
[70,13]	1,040 ms	5.42%	5.4%	5.91%	6.2%

The shape is the paper's: the first 480 ms of look-ahead buys most of the gain, 7.91 to 5.79% for the transducer, and the next 560 ms buys 0.37 points more. At the long look-aheads the measurement and the card agree within 0.3 points; at 0 and 80 ms the measured error is 0.2 to 1.2 points higher. Both decode greedily, but the card's set is the full test-other while mine is a 200-utterance subset, so the levels at small look-ahead are not comparable; the shape is.

Common misconceptions

"Causal attention makes an encoder streaming." Convolutions and the subsampling front end also see the future. The icefall streaming Zipformer reads 130 ms past every chunk because of its front end alone. Check every layer type, not only the attention.

"Latency equals chunk size." Inside a chunk the look-ahead runs from the chunk length minus one step down to zero; papers report the average, model cards often the worst case, and neither includes compute. U2 and U2++ call the same setting 640 ms and 320 ms. Always ask which.

"A right context of 2 is tiny." In a band-masked model it is 2 steps per layer, and it adds up: 17 layers × 2 steps × 80 ms is 2.7 s. Chunk-aware masking exists to stop that.

Try it

The code for this chapter is in code/ (see code/README.md, chapters 5 and 9).

CPU track

1. Run the icefall streaming Zipformer through sherpa-onnx on a 16 kHz file, chunk by chunk, and print the partial result after every chunk (code/ch03_latency/harness_demo.py does this). Note when the first word appears relative to when it was spoken. Over 200 utterances per test set, the first token appeared 631 ms (English test-clean, p50) and 599 ms (Mandarin, p50) after speech started (code/ch09_measure/results/).
2. Draw the attention mask for the chunk-aware style from the NeMo code above for a 12-step sequence and a few att_context_size values, and check by hand which steps each step can see.

GPU track

Run NeMo's cache-aware FastConformer on the RTX 4090 at several att_context_size settings, on the same fixed test subset, and plot WER against the chunk-start look-ahead (code/ch09_measure/run_gpu_all.sh; the table above is its output). Compare its shape with the paper's table above.

English and Mandarin

Repeat the CPU track on a Mandarin recording with the Mandarin streaming Zipformer. Its output is characters, and with greedy decoding a finished character, once shown, never changes. Rarer characters are spelled as two or three byte tokens, though, and show as a replacement character (�) until they are complete; chapter 6 counts how often, and explains why English partials look less stable.

Key terms

• subsampling: the front-end convolutions that turn 10 ms feature frames into longer encoder steps (40 or 80 ms).
• attention mask: which encoder steps may attend to which.
• regular look-ahead: a band mask with a fixed right context per layer; its look-ahead adds up across layers.
• chunk-aware look-ahead: attention within a chunk and to earlier chunks only; look-ahead does not grow with depth.
• att_context_size: [left, right]: NeMo's setting; the chunk is right + 1 steps.
• encoder-induced latency (EIL): an encoder's average algorithmic latency; for a chunk-aware model, papers use half the worst-case look-ahead.
• cache: the per-stream intermediate values a streaming encoder keeps between chunks.
• buffered streaming: running an offline model on overlapping windows; simple, but recomputes and mismatches training.
• dynamic chunk training: sampling the chunk size per batch so one model runs at many latencies.

Sources

• Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition (cache-aware FastConformer), arXiv:2312.17279: https://arxiv.org/abs/2312.17279
• NeMo v3.0.0 source (nemo/collections/asr/modules/conformer_encoder.py; examples/asr/conf/fastconformer/cache_aware_streaming/) and v2.7.3 docs (docs/source/asr/models.rst): https://github.com/NVIDIA/NeMo/tree/fd6a877539710e2b98f28c43272ff81312f83417, https://github.com/NVIDIA/NeMo/tree/1d4ee423806d461f9146ae982f9da8eb32495ae7
• Fast Conformer with Linearly Scalable Attention, arXiv:2305.05084: https://arxiv.org/abs/2305.05084
• Conformer, Interspeech 2020, arXiv:2005.08100: https://arxiv.org/abs/2005.08100
• Emformer, arXiv:2010.10759: https://arxiv.org/abs/2010.10759
• Zipformer, arXiv:2310.11230: https://arxiv.org/abs/2310.11230
• icefall, commit 9c96676 (egs/librispeech/ASR/zipformer/): https://github.com/k2-fsa/icefall/tree/9c966762e058bd7ba79d9da71e42fc06b64b703e
• U2, arXiv:2012.05481; U2++, arXiv:2106.05642: https://arxiv.org/abs/2012.05481, https://arxiv.org/abs/2106.05642