15. Serving streaming ASR

What you will learn

• What a streaming recognizer must keep for each open stream, and how much memory that is.
• How a server batches many streams that arrive at different times into one model call, with sherpa-onnx's WebSocket server and NeMo's slot-based cache manager as worked examples.
• What exporting a streaming model to ONNX freezes, and how partial and final results travel over the wire.
• How many streams one machine sustains: published numbers, and the book's own on one CPU and one RTX 4090.

Before you start

• Chapters: 3 (latency, percentiles, concurrency), 5 (caches, att_context_size), 6 (partial and final results), 7 (endpointing), 9 (measurement).
• Tools: the ssh-serve environment for the CPU track; for the GPU track, the NeMo environment pinned in code/serve/env/ssh-serve-nemo.lock.txt and the GPU (code/serve/README.md).

A stream is state

An offline recognizer is a function: audio in, text out, nothing kept. A streaming recognizer is not. Between two chunks of one stream it must keep:

• the samples that arrived but do not yet fill a feature window, and the feature frames not yet consumed by the encoder (chapter 2);
• the encoder caches: the attention keys and values and the convolution history of every layer (chapter 5);
• the decoder state: the last tokens for the transducer's prediction network, the beam if there is one;
• the text so far, and the endpointer's count of trailing blanks (chapter 7).

For the GPU track's cache-aware FastConformer (17 layers, model width 512, att_context_size [70,13], convolution kernel 9), NeMo allocates the caches as three tensors per batch: cache_last_channel of shape (layers, batch, 70, 512), cache_last_time of shape (layers, batch, 512, 8), and the fill level cache_last_channel_len (conformer_encoder.py, lines 1087-1125). Per stream that is 17 × 70 × 512 = 609,280 attention values plus 17 × 512 × 8 = 69,632 convolution values: 678,912 values, 2.7 MB in fp32 or 1.4 MB in bf16. The cache-aware paper notes that the attention cache "grows from zero up to the past context size" while the convolution cache is fixed (Noroozi et al., arXiv:2312.17279, Sec. 3); NeMo allocates the maximum up front and tracks the fill level. The GPU serving measurement below uses Nemotron 3.5 ASR, which is larger: 24 layers, width 1,024, a left context of 56 frames. Its cache is 24 × 56 × 1,024 + 24 × 1,024 × 8 = 1,572,864 values per stream, 6.3 MB in fp32 or 3.1 MB in bf16 (read from the model's own model_config.yaml).

So a server for streaming ASR is a state store with a model attached. Its jobs are to keep each stream's state, to notice when a stream has enough new audio for a step, and to run as many ready streams as possible in one model call.

The sherpa-onnx WebSocket server

sherpa-onnx's C++ server (v2.0.1) is small enough to read in an afternoon and shows every piece.

• One stream per connection. On a connection's first message the server creates an OnlineStream and stores it in a map keyed by the connection (online-websocket-server-impl.cc, lines 72-74). The client sends binary frames of float32 samples and finally the text "Done", after which the server appends 0.8 s of zeros and marks the input finished.
• A scan every 10 ms. A timer visits all connections (loop_interval_ms = 10). A stream is ready when it holds more unprocessed feature frames than one encoder window: GetNumProcessedFrames() + ChunkSize() < NumFramesReady() (online-recognizer-transducer-impl.h, lines 250-251). For the English Zipformer of chapters 3 and 9 the window is 45 feature frames and the step is 32, so a stream becomes ready every 320 ms of audio, once 130 ms of look-ahead beyond the step has arrived (chapter 3). The bilingual model of the serving measurement below, an older Zipformer, says T: 39 and decode_chunk_len: 32 in its ONNX metadata: the same 320 ms step with 70 ms of look-ahead beyond it.
• Batched steps. Ready streams go onto a queue. Each of three work threads pops up to five (max_batch_size = 5), stacks their feature windows into one [n, 45, 80] tensor ([n, 39, 80] for the bilingual model) and their encoder states into one set of tensors, runs the encoder and the transducer decoder once, and unstacks the new states back into the streams (DecodeStreams, lines 288-359). The English streaming Zipformer keeps six cache tensors per encoder layer (key, nonlinear attention, two values, two convolutions) plus two global ones (the bilingual model has seven kinds per encoder stack, 35 cache inputs in all); stacking concatenates the attention caches on one axis and the convolution caches on another.
• Results. After each step the server sends one JSON message per stream with the text, tokens, timestamps, is_final and segment. On an endpoint (chapter 7's rules, on by default here) it sets is_final: true, resets the stream and increments segment.

Two details differ in the Python server of the same release (python-api-examples/streaming_server.py): its messages carry text and segment but no is_final, so a client detects a final result by segment changing; and its --max-wait-ms help text says the batcher waits to fill a batch, while the code only sleeps when the queue is empty and otherwise takes whatever is there.

The real-time condition

Each stream produces one step's worth of audio every 320 ms. With W work threads, batches of at most B streams, and a batched step that takes t milliseconds, the server keeps up while

N ≤ W × B × 320 / t

streams, roughly, ignoring the 10 ms scan. With the defaults (3 threads, batches of 5) and a step of 40 ms, that is 120 streams. The step time t depends on the batch size and the CPU, and is what you measure.

Streams that start and stop at different times

Batching ten streams that all start at time zero is easy. Real streams start and stop whenever users do. NeMo's cache-aware example script batches files in lockstep and calls itself a simulation ("This script can be used to simulate cache-aware streaming", examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py). NeMo v3.0.0 also ships a pipeline for real arrival patterns, built on a slot-based cache manager (context_manager.py):

• one cache tensor covers num_slots streams (256 by default), so memory is allocated once;
• a new stream takes a free slot, or the server refuses it (raise RuntimeError("No free slots available"));
• each step gathers the ready streams' slots (self.cache_last_channel[:, slot_ids, :, :]), runs one batched step, and writes the new caches back with index_copy_;
• a finished stream's slot is zeroed with index_fill_ and freed.

With 2.7 MB of fp32 caches per stream, 256 slots of the FastConformer cost about 695 MB before any activation or weight; for Nemotron 3.5 ASR, at 6.3 MB per stream, about 1.6 GB.

Export freezes the chunk

To serve a model outside its training framework, you export it, usually to ONNX. A streaming encoder must be exported with its caches as explicit inputs and outputs, so the server can carry them between calls. icefall's streaming export names every cache as an input (cached_key_0, ...) and every updated cache as an output (new_cached_key_0, ...), and makes only the batch axis dynamic (export-onnx-streaming.py, lines 545-660). The time axis is fixed at export: for --chunk-size 16, the file says "decode_chunk_len": 32 and "T": 45. A different chunk size means a different export. NeMo's exporter keeps the cache inputs only when cache_support is set; without it, the cache inputs are removed and the exported model is offline (asr_model.py, lines 347-352).

The chunk is also frozen by training. The cache-aware paper states that "each model is trained for a specific latency and supporting multiple latencies need training of multiple models", and its remedy is training with a random chunk size per batch (arXiv:2312.17279, p. 6). That is why the GPU track's FastConformer can switch between [70,0], [70,1], [70,6] and [70,13] at run time: it was trained on all four.

State across requests in Triton

NVIDIA Triton Inference Server handles stateful models with its sequence batcher: "the sequence batcher should be used for stateful models where a sequence of inference requests must be routed to the same model instance", and it sends control signals for sequence start, end, ready and correlation ID (Triton batcher docs, v2.73.0). With implicit state, Triton stores the caches itself: "The output state provided by the model in the i-th request in the sequence will be used as the input state in the i+1-th request" (implicit_state_management.md). Two strategies batch the sequences: direct gives each sequence a dedicated batch slot, needed when the model keeps per-slot state internally; oldest "uses the dynamic batcher to batch together multiple inferences from different sequences". k2's Triton recipe for the streaming Zipformer uses oldest with implicit state, up to 1,024 candidate sequences and a 5 ms queue delay (k2-fsa/sherpa triton encoder config template).

Partial and final results on the wire

Every streaming protocol needs to say which results may still change. sherpa-onnx uses is_final (C++ server) or a change of segment (Python server). NVIDIA Riva's gRPC protocol returns partial results only when asked: "If true, interim results (tentative hypotheses) may be returned as they become available (these interim results are indicated with the is_final=false flag)" (riva_asr.proto, lines 241-251, r2.19.0). A Riva result with is_final=true will not be revised, partial results carry a stability between 0 and 1, and every result carries audio_processed in seconds. Riva also has two endpointing windows: stop_history ends an utterance, and a shorter stop_history_eou emits "a partial transcript with stability of 1" first, so that a downstream system can start work early.

Published capacity numbers

Two vendors publish stream counts for a stated setup.

Riva on an A100 (Riva 2.15.0, Conformer-CTC, English, 160 ms chunks, n-gram language model; audio sent at real-time pace, partial results off):

Streams	Average latency	p90	p95	p99	RTFx
1	13 ms	12.8 ms	13 ms	40 ms	0.999
128	86 ms	100 ms	220 ms	360 ms	126.5

The page gives the maximum as 218 streams with the language model and 223 with greedy decoding (Riva ASR performance, fetched 2026-10-29). At 128 streams the average response comes within one 160 ms chunk, but the p99, 360 ms, is more than two chunks: a few streams fall behind while the average keeps up. RTFx is 126.5 for 128 streams, so the streams run at 0.99 of real time each. The page describes its latency as "the overall latency of all returned responses", and its command turns partial results off (--interim_results=false), so these are the latencies of final results, not a word's emission delay.

Nemotron 3.5 ASR on an H100: "At the lowest-latency 80ms setting, Nemotron sustains ~17× more concurrent streams (240 vs. 14); at the 1120ms setting it sustains 6× more (2,400 vs. 400)", against a buffered Parakeet RNN-T 1.1B, with latency defined as "the median final-token latency at a given level of concurrency" (model card, revision ea30d66). At 80 ms chunks, 240 streams are 3,000 encoder steps per second; at 1,120 ms, 2,400 streams are about 2,143. A chunk 14 times larger buys 10 times more streams but no more steps per second, which suggests a large fixed cost per step at small chunks. The card does not say how the batches were formed.

Measured: the book's two tracks

The book's load generator (code/serve/ch15/loadgen.py) opens N concurrent streams, sends each a test utterance in chunks of the server's step size (160 ms on the CPU track) at real-time pace, and records when every partial and final result arrives. The servers are sherpa-onnx's Python streaming server with the CPU track's bilingual Zipformer, unchanged in its batching and decoding but with two fields added to each message so the load generator can time it (code/serve/ch15/sherpa_server.py), and a small GPU server for Nemotron 3.5 ASR built on per-stream caches and a batched step loop (code/serve/ch15/nemo_server.py).

CPU track. The server ran the bilingual English-Mandarin Zipformer (int8 encoder) with its batch limit at 3 streams, two ONNX Runtime threads and endpointing off, on the i5-12600K, whose 1-minute load average was 3.0 to 3.8 (of 16 threads) at the start of each level. Each concurrency level ran for 60 s, with the streams walking through the book's 40 English and 40 Mandarin test utterances from different starting points at real-time pace (code/serve/results/ch15_sherpa_cpu_b3t2.json). Partial latency is the time from sending a chunk to receiving the result computed from it: queueing plus compute, not the model's look-ahead. Final latency is from sending the end-of-input message to receiving the final result.

Concurrent streams	Partial latency p50 / p95	Final latency p50 / p95	Server CPU (cores)
1	19 / 22 ms	19 / 40 ms	2.7
4	18 / 26 ms	20 / 45 ms	3.2
16	23 / 48 ms	24 / 55 ms	3.6
32	37 / 85 ms	42 / 103 ms	4.0
48	884 / 1,824 ms	1,446 / 2,438 ms	4.1

Up to 16 streams the server's latency barely moves. At 32 the median partial result takes 37 ms and the p95 85 ms, still well inside one 160 ms chunk period. At 48 it collapses: the median partial result comes 884 ms after its audio, and the latency kept growing during the run, by about 183 ms per second of audio (median over sessions), which is the sign of a server that cannot keep up. The CPU column counts threads that are busy without proportional work (2.7 cores at one stream, whose compute takes about 19 ms per 160 ms chunk), so it does not show where the parallelism ran out. The knee of this configuration is between 32 and 48 streams. Error rates did not change with load (6.9% WER and 4.5 to 4.9% CER from 16 streams up); a server that falls behind delivers the same text later.

With a tuned configuration (batches of up to 16 streams, 4 threads, 2 decoding threads) the same CPU server kept up with 64 streams (partial p95 101 ms) and fell behind at 96 (partial p95 1,131 ms), using 8 to 11 cores (code/serve/results/ch15_sherpa_cpu_b16t4p2.json).

GPU track. The book's Nemotron 3.5 ASR server keeps every session's caches and decoder state and runs one loop that steps all ready streams as one batch, on one RTX 4090 in fp32 (code/serve/ch15/nemo_server.py; results code/serve/results/ch15_nemo_gpu_*.json). Its final text matched NeMo's own cache-aware loop on 20 of 20 test utterances at 80 ms, 19 of 20 at 320 ms (the other differs only by a final full stop) and 20 of 20 at 560 ms (nemo_server.py --selftest 20 --context-right r, r = 0, 3, 6). Nemotron is trained for chunks of 80, 320, 560 and 1,120 ms (att_context_size [56, r] with r = 0, 3, 6, 13), not for the running example's 160 ms, so the server refuses untrained settings and the GPU track is measured at 80, 320 and 560 ms. A server keeps up when its partial latency stops growing during the run (the median slope of latency against audio position is near zero); past that point latency grows by tens to hundreds of milliseconds per second.

Chunk	Highest level that kept up	Partial latency p95 there	Next level	One batched step (median)
80 ms	96 streams	112 ms	128 streams: fell behind (drift +51 ms per s, p95 758 ms)	12.3 ms for 1 stream, 23.5 ms for 32
320 ms	320 streams	555 ms	384 streams: fell behind (drift +12 ms per s, p95 1,306 ms)	14.6 ms for 1, 49.1 ms for 64, 161 ms for 256
560 ms	384 streams	648 ms	512 streams: out of GPU memory, level not completed	62.3 ms for 64, 199 ms for 256
320 ms, bf16	384 streams	600 ms	448 streams: server crashed, level not completed	58.9 ms for 128, 128 ms for 320

Batching is what buys the capacity. A step for 256 streams cost 161 ms, 11 times a step for one stream, for 256 times the work. Nobody configures the batch size: once the step loop is busy, the streams that become ready while a step runs form the next batch, so batches grow with load, up to 384 streams in one step at 320 ms in fp32. A bigger chunk lets a server carry more streams, because each stream needs a step less often: from 80 ms to 320 ms the highest level that kept up rose from 96 to 320 streams, the same direction as the H100 card above. At 560 ms the server kept up at 384 streams; the next level ran out of memory, so where it falls behind was not measured. Two prices come with it. The partial latency at the highest level that kept up is 112 ms at 80 ms chunks and 555 to 648 ms at the larger ones (a stream waits for the batch it falls into), so "keeps up" is not "fast"; and the 560 ms run kept up at 384 streams with a p95 of 648 ms, a little more than one chunk period. Two levels did not complete. At 512 streams and 560 ms the server ran out of GPU memory. At 448 streams in bf16 at 320 ms the server crashed with a CUDA illegal-memory-access error. In both, every session of the level errored, so the level has no full 60 s window and does not count as a capacity point. The card's uncorrectable-ECC counter read 0 before and after the sweeps, and I did not trace the cause of the crash. GPU memory grew from 3,358 MiB at one stream to 13,266 MiB at 256 (320 ms, fp32), and the server used at most 1.5 CPU cores.

Under load the text did not change. At 320 ms in fp32, WER on the English test utterances stayed at 1.6 to 2.2% and CER on the Mandarin ones at 16.8 to 17.9% from 8 streams up; at each other chunk size the rates also stayed close to flat with load (80 ms: English 3.0 to 3.6%, Mandarin 15.6 to 17.6%; 560 ms: English 2.0 to 2.1%, Mandarin 14.7 to 14.9%). The Mandarin rate is Nemotron's weak Mandarin of chapter 9.

Common misconceptions

"A streaming server is a model server with WebSockets." It is a state store: every stream's caches live on the server between chunks, and the batching logic is about state, not requests.

"Average latency below the chunk period means it keeps up." Riva's A100 table has an average of 86 ms and a p99 of 360 ms at 128 streams of 160 ms chunks.

"I can change the chunk size of an exported model." The ONNX file fixes it, and the model was trained for specific chunks. Only a model trained for several latencies can switch.

Try it

CPU track

1. Fetch sherpa-onnx's server files once with code/serve/ch15/fetch_vendor.sh, then start the book's copy of its Python server (code/serve/ch15/sherpa_server.py; code/serve/README.md has the command lines) with endpointing on, and stream one utterance with pauses through code/serve/ch15/loadgen.py. Watch segment change at each pause: this server has no is_final.
2. Run code/serve/ch15/sweep.py as its docstring shows. It starts the server, runs the load generator at each N and collects the results. Find the N at which the final latency's p95 exceeds two chunk periods.
3. Change --max-batch-size and rerun one point. Explain the change with the real-time condition above.

GPU track

code/serve/ch15/run_gpu_sweeps.sh does the same for Nemotron 3.5 ASR on the GPU. Compare the knee with the H100 card's 240 streams at 80 ms, and list every difference between the two setups.

English and Mandarin

The CPU track's server model is bilingual. Run the load generator with the Mandarin test utterances and then the English ones at the same N. The step time does not depend on the language; check whether the latency does.

Key terms

• stream state: everything a server keeps for one open stream between chunks.
• ready stream: one with enough new audio for the next encoder step.
• batched step: one model call over the stacked states of several streams.
• slot: a fixed position in a preallocated cache tensor, assigned to one stream.
• sequence batcher: Triton's scheduler for stateful models; implicit state lets Triton store the caches.
• correlation ID: the identifier that ties requests to one sequence.
• is_final, segment, stability: protocol fields that mark final results, utterance boundaries and how likely a partial result is to change.

Sources

• sherpa-onnx v2.0.1 (commit af2d8d8): sherpa-onnx/csrc/online-websocket-server-impl.{h,cc}, online-websocket-server.cc, online-recognizer-transducer-impl.h, online-zipformer2-transducer-model.{h,cc}, online-recognizer.cc, endpoint.h, python-api-examples/streaming_server.py: https://github.com/k2-fsa/sherpa-onnx/tree/af2d8d81dbfa61c4cea4c568da78f3982edca005
• NeMo v3.0.0 (commit fd6a877): nemo/collections/asr/modules/conformer_encoder.py, nemo/collections/asr/inference/utils/context_manager.py, nemo/collections/asr/models/asr_model.py, examples/asr/asr_cache_aware_streaming/speech_to_text_cache_aware_streaming_infer.py, examples/asr/conf/fastconformer/cache_aware_streaming/fastconformer_transducer_bpe_streaming.yaml: https://github.com/NVIDIA/NeMo/tree/fd6a877539710e2b98f28c43272ff81312f83417
• Noroozi et al., "Stateful Conformer with Cache-based Inference for Streaming Automatic Speech Recognition", arXiv:2312.17279.
• icefall (commit 9c96676): egs/librispeech/ASR/zipformer/export-onnx-streaming.py: https://github.com/k2-fsa/icefall/tree/9c966762e058bd7ba79d9da71e42fc06b64b703e
• Triton Inference Server v2.73.0 documentation: docs/user_guide/batcher.md, docs/user_guide/implicit_state_management.md.
• k2-fsa/sherpa (commit 497a51f): triton/model_repo_streaming_zipformer/encoder/config.pbtxt.template.
• nvidia-riva/common r2.19.0 (commit 766a50d): riva/proto/riva_asr.proto; NVIDIA Riva ASR performance page.
• NVIDIA model card nvidia/nemotron-3.5-asr-streaming-0.6b (revision ea30d66).