Appendix D. Model zoo, dated 2026-10-29¶
The chapters explain mechanisms; this appendix lists models, which change faster. Every entry names the chapter that explains how it streams; Appendix C gives the exact revision of every model the book ran or read, so you can find exactly that model again. Numbers are the ones the chapters cite, with their conditions; none of them compares models on the same test unless the row says so.
Streaming recognition¶
| Model | Design | Streaming setting | A published number, with its conditions | Chapter |
|---|---|---|---|---|
| sherpa-onnx streaming Zipformer en 2023-06-26 (CPU track) | Zipformer transducer, icefall | chunk 16 (320 ms) + 130 ms look-ahead, left 128 | measured in chapter 9: LibriSpeech test-clean, 200 utterances, greedy | 3, 9 |
| sherpa-onnx streaming Zipformer multi-zh-hans 2023-12-12 (CPU track) | Zipformer transducer | chunk 16, left 128 | measured in chapter 9: AISHELL-1 test, 200 utterances | 9 |
| icefall streaming Zipformer, LibriSpeech (66M parameters) | Zipformer transducer | chunk 320 ms / 640 ms | test-other 7.79% / 7.16% WER, greedy, chunk-wise (RESULTS.md) | 9 |
| WeNet U2++ Conformer, AISHELL-1 | CTC + attention rescoring | chunk 16 (640 ms) | 5.81% CER first pass, 5.05% final (README) | 6, 9 |
| NVIDIA stt_en_fastconformer_hybrid_large_streaming_multi (GPU track), revision ae98143 | cache-aware FastConformer, transducer + CTC | [70,13], [70,6], [70,1], [70,0] |
LibriSpeech test-other 5.4 / 5.7 / 6.4 / 7.0% transducer WER (card) | 3, 5, 9 |
| NVIDIA nemotron-3.5-asr-streaming-0.6b (GPU track), revision ea30d66 | cache-aware FastConformer, 24 layers | [56, r], chunks 80 ms to 1,120 ms |
English 9.43% at 80 ms, 7.91% at 1,120 ms; 19-locale average 10.38% / 8.84% (card) | 9, 15 |
| Kyutai stt-1b-en_fr / stt-2.6b-en | delayed streams over Mimi | 0.48 s / 2.56 s delay | 2.6B: 6.4% average WER on the paper's English sets (arXiv:2509.08753) | 8 |
| Mistral Voxtral-Mini-4B-Realtime-2602 | causal encoder + LLM decoder, 12.5 Hz | delay 80 ms to 1.2 s in 80 ms steps, or 2.4 s; 480 ms recommended | En-Short 8.47% at 480 ms (arXiv:2602.11298) | 8 |
| Moonshine v2 | sliding-window encoder, 80 ms look-ahead | — | Medium: 258 ms response latency on an Apple M3 (arXiv:2602.12241) | 8 |
| Whisper large-v3 with Whisper-Streaming or SimulStreaming | offline AED, wrapped | minimum chunk 1.0 s / 1.2 s | Whisper-Streaming large-v2: 3.62 s latency at 1 s chunks, A40 (arXiv:2307.14743) | 8 |
Streaming synthesis¶
| Model | Design | Streams by | A published number, with its conditions | Chapter |
|---|---|---|---|---|
| Kokoro-82M v1.0, v1.1-zh (CPU track), revisions f3ff357, 01e7505 | StyleTTS 2 + iSTFTNet, non-autoregressive | text segment (paragraph by default) | measured in chapter 12: CPU, sentence mode, English p50 1,351 ms TTFA | 10, 12 |
Qwen3-TTS-12Hz-1.7B-CustomVoice (GPU track), revision 0c0e305 |
backbone + multi-token prediction over a 12.5 Hz codec | 4-frame packets; served by vLLM-Omni | 101 ms first packet at one request, internal engine, unnamed device (arXiv:2601.15621) | 11, 12, 16 |
| CosyVoice 2 0.5B, Fun-CosyVoice 3 0.5B | token LM + chunked flow matching + HiFT | 25-token first hop in the released Python code | CosyVoice 2: 220 ms first chunk at one request, one L20, Triton/TensorRT-LLM runtime with a 15-token first chunk (repo docs) | 12, 14 |
| Kyutai tts-1.6b-en_fr | delayed streams, 32 Mimi codebooks | 80 ms frames, audio 16 steps behind text | 150 ms at batch 1, one H100 (arXiv:2509.08753) | 12 |
| Moshi | 7B temporal + depth transformer over Mimi | 80 ms frames | 160 ms theoretical, 200 ms in practice on an L4 | 12, 18 |
| FireRedTTS-1S | LM + flow matching, or LM + acoustic LM | 1 s chunks / frames | under 300 ms at RTF 0.1, or under 150 ms at RTF 0.3, one L20 | 12 |
| F5-TTS | flow matching, non-autoregressive | whole text batch, then sliced | RTF 0.15 at 16 steps on an RTX 3090 (arXiv:2410.06885) | 10 |
| XTTS-v2 | GPT over a 21.53 Hz VQ-VAE + HiFi-GAN | re-decode every 20 tokens + crossfade | — | 12, 13 |
| Orpheus 3B | Llama over flattened SNAC codes | 4-frame sliding window | "~200ms streaming latency" (README) | 11, 16 |
Turn detection and VAD¶
| Model | Design | Number, with conditions | Chapter |
|---|---|---|---|
| Silero VAD v6.2.3 | small network on 32 ms windows | — | 7 |
| Smart Turn v3 (pipecat-ai/smart-turn-v3, revision f766f81) | Whisper Tiny encoder + linear head, 8M parameters | 12.6 ms on an AWS c7a.2xlarge (Daily) | 7, 18 |
| LiveKit turn detector (text) | Qwen2.5-0.5B fine-tune, INT8 ONNX | "~50-160 ms" per turn on a CPU | 7 |
Codecs¶
| Codec | Frame rate | Codebooks | Causal | Chapter |
|---|---|---|---|---|
| EnCodec 24 kHz | 75 Hz | up to 32 of 1,024 | yes | 11 |
| DAC 44.1 kHz / 24 kHz | 86.13 Hz / 75 Hz | 9 / up to 32 of 1,024 | no | 11 |
| SNAC 24 kHz | 11.72 / 23.44 / 46.88 Hz | 1 + 2 + 4 of 4,096 | no | 11 |
| Mimi | 12.5 Hz | 8 to 32 of 2,048 | yes | 11, 13 |
| Qwen3-TTS 12 Hz tokenizer | 12.5 Hz | 16 (1 of 4,096, 15 of 2,048) | yes | 11 |
| CosyVoice 2 and 3 tokenizer | 25 Hz | 1 of 6,561 (FSQ) | — | 11 |