8. Whisper and LLM-based ASR in streaming

What you will learn

• The difference between wrapper streaming, which re-runs an offline model on a growing buffer, and native streaming, where the model itself was trained to emit as audio arrives.
• How the two wrapper policies in common use decide what to commit: LocalAgreement (Whisper-Streaming) and AlignAtt (Simul-Whisper, SimulStreaming).
• What native streaming models built on language models look like (Delayed Streams Modeling, Voxtral Realtime, Speech ReaLLM, Moonshine v2), with their published delays and accuracies.
• How to choose between a wrapper and a native model for a given latency budget.

This chapter is optional on a first read. Chapters 9 to 18 do not depend on it.

Before you start

• Chapters: 3 (latency vocabulary), 4 (attention-based encoder-decoders do not stream), 6 (partial and final results).
• Tools: plain Python 3 for Try it; no extra packages.

Why Whisper does not stream

Whisper is an attention-based encoder-decoder (chapter 4) trained on 30 s windows: "We break audio files into 30-second segments" (Radford et al., arXiv:2212.04356, Sec. 2.1). The code fixes the window: CHUNK_LENGTH = 30, 480,000 samples, 3,000 mel frames, and anything shorter is zero-padded to that length before the encoder sees it (whisper/audio.py, lines 13-18 and 65-88). Its encoder attends over the whole window, and its decoder writes the transcript after the encoder has seen everything. Neither half can produce a word before the window is complete.

Two consequences follow. A 2 s utterance costs a full 30 s encoder pass. And the only way to get text early is to call the model again and again on whatever audio has arrived so far, and decide which of its outputs to trust. That is wrapper streaming.

LocalAgreement: commit what two passes agree on

Whisper-Streaming (Macháček et al.) keeps an audio buffer, runs Whisper on the whole buffer every time a new chunk of audio arrives, and commits only the words that two consecutive passes agree on. The policy, LocalAgreement-n, "outputs the longest common prefix of the model on n consecutive source chunks", and "LocalAgreement with n = 2 was the most effective policy" (arXiv:2307.14743, Sec. 2). Committed text is never revised: "Changes to the transcription in the confirmed portion are disregarded" (Sec. 3).

Here is the policy on four made-up updates, computed by code/ch08_whisper/la2_trace.py:

Update	Hypothesis on the whole buffer	Committed now	Still pending
1 (1 s of audio)	Thank you, Mr.	—	Thank you, Mr.
2 (2 s)	Thank you, Mr. President.	Thank you, Mr.	President.
3 (3 s)	Thank you, Mr. President. Today	President.	Today
4 (4 s)	Thank you, Mr. President. Today, I	—	Today, I

Two things are visible. Every word needs two agreeing passes, so a word spoken in one chunk is committed at the earliest one chunk later. The paper says exactly this: "the average computationally unaware latency is approximately twice the chunk size. This is expected because we use local agreement of two consecutive updates." And words match as exact strings: in update 4, "Today," with a comma no longer matches "Today", so nothing is committed. The real code compares words the same way (HypothesisBuffer.flush in whisper_online.py, lines 396-417).

Then add the computation. Every update re-runs Whisper from the start of the buffer, and the buffer grows until it is trimmed. The paper's measurements on an A40 GPU, large-v2 with faster-whisper, English (arXiv:2307.14743, Table 3). "Unaware" assumes Whisper computes instantly, so it shows the policy's own wait; "aware" includes the real computation:

Minimum chunk size	WER (aware)	Average latency, unaware	Average latency, aware
offline	7.9%	—	—
0.5 s	8.5%	1.02 s	3.27 s
1.0 s	8.1%	1.91 s	3.62 s
2.0 s	8.0%	3.73 s	5.45 s

The unaware column is the "approximately twice the chunk size" of LocalAgreement-2; computation adds 1.7 to 2.3 s more.

The abstract's "3.3 seconds latency" matches the 0.5 s row (3.27 s), not the 1.0 s row (3.62 s) that the text pairs it with; the same sentence's German and Czech figures do match the 1.0 s rows. The table is what was measured.

The code also differs from the paper in two places that change behavior, and the code is what runs: the buffer is trimmed at 15 s by default (--buffer_trimming_sec 15, lines 778-779), not "around 30 seconds"; and the prompt carried into the next pass is the last 200 characters of committed text (while p and l < 200: # 200 characters prompt size, line 470), not 200 words. Its own README now says it "is becaming outdated, replaced by SimulStreaming" (typo in the original).

AlignAtt: stop where the attention reaches the edge

The second policy decides with one pass instead of two. Whisper's decoder attends to encoder frames while it writes each token, and the most attended frame is a rough guess of where in the audio that token is. AlignAtt, from simultaneous translation, uses this: "we stop the emission as soon as we find a token that mostly attends to one of the last f frames" (Papi et al., arXiv:2305.11408, Sec. 2). If the decoder is looking at the very end of the buffer, the word there may be cut off, so it waits for more audio.

Simul-Whisper (Wang et al.) applied this to Whisper with a margin of 12 frames, 240 ms, and added a small module that detects and deletes a word truncated at the chunk boundary (arXiv:2406.10052, Secs. 2.1-3.3). At a 1 s chunk, its abstract reports "an average absolute word error rate degradation of only 1.46%". That number is for Whisper Medium; in the same table the degradation is 7.68 points for Base, 3.23 for Small and 2.34 for Large-v2, against 10.59 (Medium) and 6.90 (Large-v2) for the LocalAgreement baseline (Table 1, averaged over four LibriSpeech and seven Multilingual LibriSpeech evaluation sets, test and dev).

SimulStreaming (Macháček et al.) combines AlignAtt with Whisper-Streaming's long-form interface, and adds large-v3, beam search and context carried across 30 s windows (arXiv:2506.17077, Sec. 3). Its command-line defaults are a 1.2 s minimum chunk (simulstreaming/whisper/whisper_streaming/whisper_online_main.py, line 35), greedy decoding (--beams 1), and a margin of 25 frames, which its help text puts at 0.5 s ("one frame is 0.02 seconds for large-v3 model") (simulstreaming_whisper.py, lines 17 and 29-31). Its README claims it is "~5-times faster" than Whisper-Streaming; neither the README nor the paper says how that was measured.

Both policies share the wrapper's basic cost: a full encoder pass over up to 30 s of audio for every update, however little audio is new.

Native streaming with language models

A native streaming model is trained to read audio and emit text as it goes, with a fixed delay between the two. Four designs show the range.

Delayed Streams Modeling (Kyutai). Audio is encoded by the Mimi codec at 12.5 Hz, 80 ms per frame, and the text stream is the same sequence shifted by a fixed number of frames. The model at time t emits the text of the audio from t minus the delay (Zeghidour et al., arXiv:2509.08753). Two models are released: kyutai/stt-1b-en_fr, with "a 0.5 second delay, and a semantic VAD", whose configuration in the delayed-streams-modeling repository says asr_delay_in_tokens = 6 (480 ms, configs/config-stt-en_fr-hf.toml); and kyutai/stt-2.6b-en, with "a 2.5 second delay", configured as 32 frames (2.56 s, configs/config-stt-en-hf.toml). On the paper's English benchmarks the 2.6B model averages 6.4% WER, against 9.4% for Whisper-Streaming with large-v3 run at the same 2.5 s delay (Sec. 4.3, Table 1) and 7.5% for offline Whisper large-v3 (Table 1), and an H100 serves "400 sequences simultaneously while being real-time". The weights are CC-BY 4.0; the code is MIT (Python) and Apache (Rust).

A fixed delay has a cost at the end of an utterance: the last 2.5 s of text come out only after 2.5 s of further audio. Kyutai's voice-agent server hides this with what it calls a flush: when the user pauses, it feeds the model ceil(delay_sec / FRAME_TIME_SEC) + 1 frames of zeros faster than real time (unmute_handler.py, lines 340-370), pushing the delayed text out at compute speed instead of waiting for real audio.

Voxtral Realtime (Mistral). A 4.4B model: a causal audio encoder (970M parameters, 50 Hz, a 15 s sliding window), an adapter that pools four frames into one so that "each downsampled audio embedding represents 80 ms", and a 3.4B decoder (arXiv:2602.11298, Table 1 and Sec. 2.2). It was trained with a delay sampled "uniformly from 80 ms to 2400 ms in 80 ms increments"; the model card recommends 480 ms and allows "any multiple of 80ms between 80 and 1200, as well as 2400". The abstract says it is "on par with Whisper" at 480 ms. That holds for English on macro averages: 8.47% against 8.39% on the English short-form sets, 7.73% against 7.97% on long-form (Table 3). On the multilingual sets it is behind: 8.72% against 8.23% on FLEURS, 15.24% against 14.25% on Common Voice. On LibriSpeech, offline Whisper is clearly better: 1.84% and 3.66% on test-clean and test-other against Voxtral's 2.08% and 5.54% at 480 ms (Table 5). The weights are Apache 2.0.

Speech ReaLLM. A small multimodal language model taught, in its title's words, "the flow of time": it emits text while audio is still arriving, and "an 80M Speech ReaLLM achieves WERs of 3.0% and 7.4% in real time" on LibriSpeech test-clean and test-other, against 3.6% and 9.4% for an RNN-T of similar size (Seide et al., arXiv:2406.09569, Table 1).

Moonshine v2. An edge model that attacks the encoder side of the problem. With a full-attention encoder, "this causes TTFT to grow linearly with utterance length as the encoder must process the entire prefix" (abstract). Moonshine v2 uses a sliding-window encoder in which four layers each look 4 frames of 20 ms ahead; the paper gives "a conservative bound" of "16 × 20 ms = 320 ms of future context" before an encoder state is final (Kudlur et al., arXiv:2602.12241, Sec. 3.2). On an Apple M3, its Medium model's response latency was 258 ms against 11,286 ms for Whisper Large v3 (Table 2). MIT licence.

Read the figure's two x-axes carefully. The left one is a measured latency. The right one is a configured delay, which chapter 3 showed to be a lower bound: at its 80 ms setting, Voxtral Realtime's measured median word emission delay was 0.77 s (arXiv:2609.18333). Compare models on the same measured quantity, on the same audio, as chapter 9 does.

Choosing

The decision comes down to three questions.

1. What latency do you need? A wrapper's word emission latency is at least about twice its chunk for LocalAgreement, plus a full encoder pass per update: seconds, in the published measurements. A native model's delay is configured, from 80 ms upwards, though what users see is larger.
2. What accuracy do you need, on what audio? Offline Whisper remains strong, and the best native models match it on average only at delays of about half a second or more.
3. What do you have to run it on? A wrapper re-encodes up to 30 s of audio for every update, which is expensive at many concurrent streams. Native models process each frame once. The book's two tracks use native streaming models for this reason: transducers, which stream by construction (chapter 4).

A wrapper is the right tool when you need Whisper's languages or robustness, latency of a few seconds is acceptable, and the number of streams is small, as with live captions for one talk. For a voice agent with a sub-second budget, start from a native streaming model.

Common misconceptions

"Whisper with a small chunk is a streaming model." It is an offline model called repeatedly. Each call encodes a 30 s window, and LocalAgreement adds about one chunk of waiting on top.

"Committed text from Whisper-Streaming can still be corrected." It cannot. The policy never revises the confirmed portion; only pending text changes.

"A 480 ms delay setting means words appear 480 ms after they are spoken." The setting is a lower bound. Measured emission delays are larger, as chapter 3 showed.

Try it

CPU track

1. Run python code/ch08_whisper/la2_trace.py. It prints the four-update table above.
2. Change the fourth hypothesis to end with "Today I" (no comma) and run it again. "Today" is now committed at update 4. Then restore it and make the second hypothesis say "Thank you Mr. President." (no comma after "you"). Only "Thank" is committed at update 2, nothing at update 3, and "you, Mr. President." only at update 4: one missing comma delayed two words by two updates and one by one.
3. Add a fifth hypothesis in which the model rewrites an earlier word ("Thank you, Mrs. President. Today, I am"). The committed words do not change; see what the function does with the rest. The book's short script then puts the whole hypothesis, committed words included, back into "pending"; that is an artifact of the simplification, which the real whisper_online.py avoids by matching on timestamps.

GPU track

The wrappers in this chapter are open source (both MIT) and run on one RTX 4090. If you want to see the re-encoding cost, run Whisper-Streaming's whisper_online.py with --min-chunk-size 0.5 and then 2.0 on one long recording, and compare its GPU time per second of audio. The book does not measure them; chapter 9 measures the native streaming models of the two tracks.

English and Mandarin

LocalAgreement compares whole words separated by spaces. Mandarin text has no spaces, so a wrapper must first split it into words or characters for the comparison. Try it: give la2_trace.py Mandarin hypotheses as lists of characters, then as lists of words, and see how much earlier characters commit.

Key terms

• wrapper streaming: re-running an offline model on a growing audio buffer and committing part of each output.
• native streaming: a model trained to emit output while audio is still arriving.
• LocalAgreement-n: commit the longest common prefix of the last n hypotheses.
• AlignAtt: stop emitting when a token's most attended encoder frame is within the last f frames.
• commit: the point at which a word becomes final and will not change.
• delay (configured): the fixed offset between audio and text a native model was trained or set to use; a lower bound on its emission delay.
• flush: feeding zeros faster than real time to push a delayed model's last words out.

Sources

• Radford et al., "Robust Speech Recognition via Large-Scale Weak Supervision", arXiv:2212.04356; openai/whisper v20250625 (commit 31243ba), whisper/audio.py: https://github.com/openai/whisper/tree/31243bad24cc746f07d4c8bfdd2d974872cb1803
• Macháček et al., "Turning Whisper into Real-Time Transcription System", arXiv:2307.14743; ufal/whisper_streaming (commit 6da90b4), whisper_online.py, README.md: https://github.com/ufal/whisper_streaming/tree/6da90b44b7e50d79695e68166d2a2c7609c75abb
• Papi et al., "AlignAtt: Using Attention-based Audio-Translation Alignments as a Guide for Simultaneous Speech Translation", arXiv:2305.11408.
• Wang et al., "Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detection", arXiv:2406.10052.
• Macháček et al., "Simultaneous Translation with Offline Speech and LLM Models in CUNI Submission to IWSLT 2025", arXiv:2506.17077; ufal/SimulStreaming (commit 077ea37): https://github.com/ufal/SimulStreaming/tree/077ea37d5ab4ff98bc567e4507f140dc4e5d5ad6
• Zeghidour et al., "Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling", arXiv:2509.08753; kyutai-labs/delayed-streams-modeling (commit 4c4f65e); kyutai-labs/unmute (commit e348e56); model cards kyutai/stt-1b-en_fr (revision 1c34c6b) and kyutai/stt-2.6b-en (revision a07aec5).
• Mistral AI, "Voxtral Realtime", arXiv:2602.11298; model card mistralai/Voxtral-Mini-4B-Realtime-2602 (revision 2769294).
• Seide et al., "Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time", arXiv:2406.09569.
• Kudlur et al., "Moonshine v2: Ergodic Streaming Encoder ASR for Latency-Critical Speech Applications", arXiv:2602.12241; moonshine-ai/moonshine (commit 234f60f).
• "Look Less, Hear Better: Jointly Rewarded GRPO for Streaming ASR", arXiv:2609.18333.