Appendix B. Glossary

Each term is defined in the chapter given in parentheses.

• acoustic echo cancellation (AEC) (chapter 17): removing the device's own speaker sound from its microphone signal.
• acoustic model (chapter 10): predicts durations, prosody and a mel spectrogram or codec tokens.
• acoustic tokens / semantic tokens (chapter 11): tokens that describe the sound, or the content.
• algorithmic latency (chapter 3): chunk plus look-ahead; the delay on an infinitely fast computer.
• aliasing (chapter 1): frequencies above the Nyquist limit folding back as false lower ones.
• AlignAtt (chapter 8): stop emitting when a token's most attended encoder frame is within the last f frames.
• alignment (chapter 4): which frame each output unit belongs to.
• async chunking (chapter 16): handing codec frames to the decoder in chunks while the talker is still writing.
• att_context_size: [left, right] (chapter 5): NeMo's setting; the chunk is right + 1 steps.
• attention encoder-decoder (AED) (chapter 4): a decoder that generates tokens while attending over all encoder frames.
• attention mask (chapter 5): which encoder steps may attend to which.
• audio continuity (chapter 16): whether a request plays without a gap longer than a threshold (0.1 s in vLLM-Omni).
• autoregressive / non-autoregressive (chapter 10): producing output one step at a time in order, or all at once.
• backbone / depth transformer (chapter 12): one large model step per frame, and a small model for the codes inside it.
• backlog (chapter 9): time between a chunk's arrival and the start of its processing.
• barge-in (chapter 18): the user speaking while the agent speaks; the agent should stop.
• batched step (chapter 15): one model call over the stacked states of several streams.
• bin (chapter 2): one output of the transform; its frequency is k times the sample rate divided by N.
• bi-streaming (chapter 10): both at once.
• bit depth (chapter 1): bits per sample; 16 bits gives 65,536 values.
• blank (CTC), ∅ (transducer) (chapter 4): the extra symbol meaning "output nothing here".
• buffered streaming (chapter 5): running an offline model on overlapping windows; simple, but recomputes and mismatches training.
• cache (chapter 5): the per-stream intermediate values a streaming encoder keeps between chunks.
• capacity knee (chapter 16): here, the lowest load at which TTFA p95 or continuity leaves its target.
• causal codec (chapter 11): one whose outputs depend only on present and past input, so it streams without waiting.
• character error rate (CER) (chapter 9): the same over characters; the standard for Mandarin.
• chunk (chapter 3): the piece of audio processed at once; the client's and the model's may differ.
• chunk-aware look-ahead (chapter 5): attention within a chunk and to earlier chunks only; look-ahead does not grow with depth.
• chunk deadline (chapter 14): TTFA plus the duration of all earlier chunks; a later arrival is an underrun.
• chunk mask (chapter 12): an attention mask that lets a frame see only its own chunk and earlier ones.
• chunk ramp (chapter 16): chunk sizes that grow gradually after the first, so that each chunk's deadline can be met.
• CMVN (chapter 2): subtracting a mean and dividing by a standard deviation per feature dimension.
• codebook (chapter 11): the learned table a quantizer picks entries from; bits per code = log₂ of its size.
• codec frame (chapter 11): one step of the codec's encoder output; frame rate in Hz.
• codec frame (chapter 17): the stretch of audio a codec compresses as one unit; Opus uses 2.5 to 60 ms.
• codec language model (chapter 12): a language model that writes codec tokens as its output.
• collapse rule (chapter 4): merge repeated symbols, then remove blanks.
• commit (chapter 8): the point at which a word becomes final and will not change.
• compute latency (chapter 3): the time to process a chunk once its audio has arrived.
• contextual biasing / hotwords (chapter 6): boosting a given list of phrases during decoding.
• correlation ID (chapter 15): the identifier that ties requests to one sequence.
• crossfade (chapter 13): blending the end of one chunk into the start of the next over an overlap.
• dBFS (chapter 1): decibels relative to the largest representable value.
• delay (configured) (chapter 8): the fixed offset between audio and text a native model was trained or set to use; a lower bound on its emission delay.
• delayed streams (chapter 12): text and audio on one clock, the audio shifted a fixed number of steps behind.
• delay penalty (chapter 6): a training term that rewards early and penalizes late label emissions in the lattice.
• discrete Fourier transform (DFT) (chapter 2): N samples in, N numbers out, one per bin; bin k measures how much of the signal repeats k times in the N samples. The FFT is its fast algorithm.
• dynamic chunk training (chapter 5): sampling the chunk size per batch so one model runs at many latencies.
• emission delay (chapter 3): from when a word is spoken to when the recognizer outputs it.
• emission delay (chapter 4): how late, relative to the audio, a model emits each label.
• emission delay (chapter 6): how late a model emits a label relative to when it was spoken.
• encoder-induced latency (EIL) (chapter 5): an encoder's average algorithmic latency; for a chunk-aware model, papers use half the worst-case look-ahead.
• endpoint delay (chapter 3): from the end of speech to the decision that the user has finished.
• endpoint delay (chapter 18): the wait after the last word before the system decides the user's turn is over.
• endpointer, endpointing (chapter 7): the decision that an utterance has ended and its result should be finalized.
• excitation (source) signal (chapter 13): the pitch-driven signal of a source-filter vocoder such as HiFT.
• false cut-off (chapter 7): an endpoint decided at a pause inside the user's utterance.
• FastEmit (chapter 6): a training regularizer that rewards emitting labels early.
• filling token (chapter 12): CosyVoice 2's token meaning "give me the next text tokens".
• final result (chapter 6): the committed text of a finished segment.
• finite scalar quantization (FSQ) (chapter 11): rounding a few dimensions to fixed levels; one large implicit codebook.
• first-token delay (chapter 3): from the start of speech to the first output token.
• flattened tokens (chapter 11): every code of every frame written as a separate language-model token.
• flush (chapter 8): feeding zeros faster than real time to push a delayed model's last words out.
• formant (chapter 2): a broad hump of energy shaped by the throat and mouth; the first two or three tell vowels apart.
• forward algorithm (chapter 4): dynamic programming that sums the probability of all alignments.
• frame (chapter 2): one short slice of audio for feature extraction, usually 25 ms.
• fricative (chapter 2): a consonant made by forcing air through a narrow gap (s, sh, f; z and v when voiced); noise spread over high frequencies, with a buzz at low frequencies added when voiced.
• full duplex (chapter 18): listening and speaking at the same time, modelling both speakers' streams together.
• fundamental frequency (f0) (chapter 1): the reciprocal of the pitch period, in Hz; for the 3.99 ms period above it is 250.8 Hz. Pitch is how high a listener hears the voice; f0 is the number measured for it.
• grapheme-to-phoneme (G2P) (chapter 10): converting written words into phonemes.
• greedy search (chapter 6): take the most probable symbol at each step.
• harmonic (chapter 2): a multiple of the voice's pitch; a voiced sound has energy at f0, 2 f0, 3 f0 and so on.
• head-of-line blocking (chapter 17): on TCP, every later byte waits for a lost one.
• hold-back (chapter 13): samples a streaming vocoder keeps until the next chunk arrives, to crossfade them.
• hop (chapter 2): the distance between the starts of consecutive frames, usually 10 ms.
• hysteresis (chapter 7): separate start and end thresholds, so a decision does not flip on every window.
• in-band FEC (chapter 17): a low-bitrate copy of an important frame carried in a later packet.
• input streaming (chapter 10): synthesis starts before all the text has arrived.
• inter-arrival time (chapter 14): time between the arrivals of consecutive chunks.
• interleaved (chapter 1): channels stored sample by sample, left then right.
• interleaving (N:M) (chapter 12): alternating N text tokens and M speech tokens in one sequence, for input streaming.
• internal language model estimation (ILME) (chapter 6): subtracting the model's own built-in language model when fusing an external one.
• is_final, segment, stability (chapter 15): protocol fields that mark final results, utterance boundaries and how likely a partial result is to change.
• jitter (chapter 17): variation in packet arrival time.
• jitter buffer (chapter 17): the buffer that turns irregular arrival into steady playout; its depth is delay.
• joiner (chapter 4): combines encoder and prediction vectors into a distribution over labels and ∅.
• left context / right context (chapter 13): frames before or after a chunk given to the vocoder and thrown away; right context must be waited for.
• little-endian / big-endian (chapter 1): least or most significant byte first.
• LocalAgreement-n (chapter 8): commit the longest common prefix of the last n hypotheses.
• log-mel feature / fbank (chapter 2): the logarithm of each mel filter's output; what the models read.
• look-ahead (chapter 3): audio after a chunk that the model must wait for before processing it.
• lossy / lossless (chapter 1): a lossy codec returns samples that differ from the input; raw PCM keeps every number.
• magnitude (chapter 2): the length of a bin's complex value; how much of that frequency is present, without the phase.
• mel filterbank (chapter 2): triangular filters spaced on a mel scale (HTK or Slaney) that pool the spectrum.
• mel scale (chapter 2): a frequency scale that is finer at low frequencies, built to follow hearing.
• MFCC (chapter 2): log-mel values followed by a discrete cosine transform, keeping the first few outputs.
• minimum silence (chapter 7): how long speech must be absent before the VAD reports an end.
• modified beam search (chapter 6): icefall's transducer beam search with one label per frame and merged equal hypotheses.
• MOS / MOS predictor (chapter 14): human mean opinion score, and models that predict it.
• native streaming (chapter 8): a model trained to emit output while audio is still arriving.
• neural audio codec (chapter 11): an encoder, quantizer and decoder trained together to turn audio into tokens and back.
• neutral tone (chapter 1): the fifth Mandarin tone, with no specific contour, on weak syllables.
• normalized autocorrelation (chapter 1): how well a stretch of signal matches itself shifted by a lag, from −1 to 1; 1 means an exact repeat.
• Nyquist limit (chapter 1): half the sample rate, the highest frequency a sampled signal can represent.
• open-loop load (chapter 16): requests sent at a fixed rate regardless of the server's progress.
• Opus (chapter 1): a lossy audio codec for speech and music, 6 to 510 kbit/s; chapter 17.
• Opus (chapter 17): the standard real-time audio codec of the web (RFC 6716).
• output streaming (chapter 10): audio leaves in chunks before generation is complete.
• packet (chapter 12): the unit of audio a streaming TTS hands over; 4 tokens, 320 ms, for Qwen3-TTS 12 Hz.
• packet (chapter 17): one or more frames plus network headers, sent as one unit.
• packet loss concealment (PLC) (chapter 17): the decoder's substitute for a missing frame.
• partial result (chapter 6): the current, revisable best guess for the utterance so far.
• path (CTC) (chapter 4): one symbol per frame; collapsing it gives a labelling.
• percentile (p50, p95, p99) (chapter 3): the value below which that share of requests fall; needs a count.
• per-utterance normalization (chapter 2): CMVN with statistics of the whole utterance; reads the future.
• pinyin (chapter 1): the standard romanization of Mandarin, in Latin letters with tone marks.
• pitch period (chapter 1): the shortest time after which a voiced waveform repeats, for example 3.99 ms in the English recording of this chapter.
• pooled WER (chapter 9): total edits over total reference words for a test set.
• pre-buffer (chapter 14): a wait after the first audio before playback starts, traded against TTFA.
• prediction network (chapter 4): the part of a transducer that reads previously emitted labels.
• prosody reset (chapter 13): intonation restarting at a boundary because the acoustic model lost context.
• ready stream (chapter 15): one with enough new audio for the next encoder step.
• real-time factor (RTF) (chapter 3): processing time divided by audio duration; RTFx is its inverse.
• receptive field (chapter 13): the span of input frames one output sample depends on.
• regular look-ahead (chapter 5): a band mask with a fixed right context per layer; its look-ahead adds up across layers.
• render quantum (chapter 17): the block of samples the Web Audio API processes at once, 128 frames by default.
• resampling (chapter 1): changing the sample rate, with filtering to prevent aliasing.
• residual vector quantization (RVQ) (chapter 11): stacked codebooks, each coding the previous ones' error.
• round-trip latency (device) (chapter 17): speaker-to-microphone delay through a loopback.
• sample (chapter 1): one measurement of the signal.
• sample rate (chapter 1): samples per second, in Hz; speech models use 16 kHz.
• Seed-TTS-eval (chapter 14): the common intelligibility and similarity protocol for zero-shot TTS.
• sentence aggregation (chapter 18): collecting LLM tokens until a sentence boundary before synthesis.
• sequence batcher (chapter 15): Triton's scheduler for stateful models; implicit state lets Triton store the caches.
• shallow fusion (chapter 6): adding an external language model's score during decoding.
• simulated pacing (chapter 9): replaying audio against a virtual clock instead of sleeping.
• slot (chapter 15): a fixed position in a preallocated cache tensor, assigned to one stream.
• snip_edges (chapter 2): Kaldi's choice between frames that fit inside the audio and frames centered on their hop.
• speaker similarity (SIM) (chapter 14): cosine similarity of speaker embeddings; depends on the embedding model.
• spectral centroid (chapter 2): the power-weighted average frequency of a sound.
• spectrogram (chapter 2): power per frequency bin per frame.
• stop release (burst) (chapter 2): the click when a closed mouth opens (p, t, k, b, d, g); a short vertical stripe. An affricate (zh, j, ch) is a stop release followed by a fricative.
• stream state (chapter 15): everything a server keeps for one open stream between chunks.
• subsampling (chapter 5): the front-end convolutions that turn 10 ms feature frames into longer encoder steps (40 or 80 ms).
• substitution, deletion, insertion (S, D, I) (chapter 9): the three edit types of an alignment.
• takeover rate (TOR) (chapter 18): in Full-Duplex-Bench, the share of test points at which a model takes the turn.
• talker / code2wav (chapter 16): the token-generating stage and the codec-decoding stage of a two-stage TTS server.
• text normalization (TN) (chapter 10): rewriting numbers, abbreviations and symbols as words.
• text normalizer (chapter 9): the transformation applied to reference and hypothesis before scoring.
• time to first audio (TTFA) (chapter 3): from the text request to the first audible sample of synthesized speech.
• time to first audio (TTFA) (chapter 14): here, from sending the request (or first text) to the arrival of the first audible sample, on the client.
• tone (chapter 1): in Mandarin, the pitch contour of a syllable, which distinguishes words; four main tones (high level, rising, dipping, falling) and a neutral one.
• trailing silence (chapter 7): in sherpa-onnx, the run of blank outputs since the last token, in seconds.
• transducer (RNN-T) (chapter 4): encoder, prediction network and joiner on a frames × labels lattice.
• turn detector (chapter 7): a model that decides whether the user has finished their turn, from text or audio.
• turn detector (chapter 18): a model that predicts from audio (and sometimes text) whether the user has finished.
• two-pass decoding (chapter 6): a streaming first pass for partials, rescored at the end of a segment for the final.
• underrun (chapter 12): the player runs out of audio before the next chunk arrives.
• underrun / stall (chapter 14): the player has nothing to play; stall is the total silence it causes.
• UPWR, UPSR (chapter 9): unstable partial word ratio and unstable partial segment ratio.
• user-perceived latency (UPL) (chapter 3): from the end of speech to the system's response; say which response.
• vocoder (chapter 10): turns a mel spectrogram into a waveform; a codec decoder does the same from codec tokens.
• vocoder (chapter 13): turns mel frames into a waveform; GAN-based (HiFi-GAN, BigVGAN) or inverse-STFT (Vocos).
• voice activity detector (VAD) (chapter 7): decides, per short window, whether audio contains speech.
• voice cascade (chapter 18): speech recognizer, language model and speech synthesizer in a row.
• voiced / unvoiced (chapter 1): a speech sound is voiced when the vocal folds vibrate while it is made; its waveform repeats. An unvoiced sound has no such vibration; an unvoiced fricative such as [s] looks like noise.
• voice-to-voice latency (chapter 18): time from the end of the user's speech to the start of the agent's speech, as heard.
• window (chapter 2): the taper multiplied into a frame (Hann, povey), and also its length.
• word error rate (WER) (chapter 9): (S + D + I) / reference words; can exceed 100%.
• word error rate (WER) / character error rate (CER) (chapter 1): edits over reference words, or over reference characters; WER is the usual metric for English and CER for Mandarin (chapter 9).
• wrapper streaming (chapter 8): re-running an offline model on a growing audio buffer and committing part of each output.