Skip to content

Preface

This handbook is for engineers who are new to streaming speech: you can write Python and have trained or run a neural network, and you now need to build, evaluate or operate a system that listens and speaks while the user is still talking. It starts from audio as a list of numbers and ends with the measured latency budget of a complete voice agent.

Good material exists at each end of this path: courses that start from acoustics and reach streaming speech recognition, and practitioner guides that measure a voice agent's latency budget with recognition and synthesis treated as black boxes. I found none that explains how a streaming text-to-speech model produces its first audio, or that carries a reader from the waveform, through the internals of both streaming recognition and streaming synthesis, to the measured latency of a whole voice system. This book tries to do both.

What is in it

The book has five parts.

  • Foundations (chapters 1 to 3). Samples, frames and features; the vocabulary of latency that the rest of the book uses: chunk, look-ahead, compute, emission delay, endpoint delay, time to first audio, percentiles.
  • Streaming recognition (chapters 4 to 9). CTC and the transducer, with their losses worked out by hand; streaming encoders and their caches; decoding, partial results and contextual biasing; endpointing, voice activity detection and turn detection; Whisper and language-model recognizers in streaming; and how to measure all of it.
  • Streaming synthesis (chapters 10 to 14). The TTS pipeline and the two meanings of streaming; neural audio codecs; the streaming acoustic models (codec language models, chunked flow matching, delayed streams); vocoders and what happens at chunk boundaries; and how to measure time to first audio, gaps and quality.
  • Serving (chapters 15 and 16). Stream state, batching and capacity for recognition and synthesis servers.
  • The real-time system (chapters 17 and 18). Transport and the client, then the whole cascade: recognition, turn detection, a language model and synthesis, with a measured budget, barge-in, and full duplex.

The appendices hold conversion tables, a glossary, the toolkits and their licences, a dated list of models, and a short guide to fine-tuning a streaming model on a small data set.

Chapter 8 (Whisper and language-model recognizers in streaming) is optional on a first read. Every other chapter builds on the ones before it.

Two tracks, two languages

Every chapter ends with a Try it section with two tracks.

  • The CPU track runs on an ordinary computer: sherpa-onnx with streaming Zipformer models for recognition, Kokoro-82M for synthesis, Silero VAD for voice activity detection.
  • The GPU track runs on one consumer GPU: NVIDIA NeMo's cache-aware FastConformer and Nemotron 3.5 ASR for recognition, Qwen3-TTS served by vLLM-Omni for synthesis.

Every example and measurement is done in English and Mandarin, because the two languages disagree in useful places: tones, word against character error rates, spaces, numerals, sentence boundaries.

Conventions

  • Every number has a source. A number from a paper cites the paper and the page or table. A number from code cites the file and line at a pinned version. A number I measured names the machine, the script that produced it and the result file it is in. The measurements in this book were made on one machine: an Intel Core i5-12600K and one NVIDIA RTX 4090 (23,028 MiB usable). A number measured on a different card is a different number.
  • Where sources disagree, the book says so, gives both values, and says which one the code you will run uses.
  • Mechanisms are in the chapters, models in the appendix. Models change every few months; the reasons a model has a given latency change much more slowly. Appendix D lists the models used here with the date the list was made.
  • Figures that show synthetic data say so in their corner. Every other figure is drawn from a data file that sits beside its script in figures/.

What you need

  • Python 3.11 and the conda environments described in code/README.md, code/tts/README.md and code/serve/README.md. Each environment has a lock file.
  • For the GPU track, one NVIDIA GPU; the measurements here used one with 23,028 MiB usable.
  • The public data sets the chapters name: LibriSpeech and LibriTTS-R for English, AISHELL-1 for Mandarin. Their licences are in Appendix C.