12. Streaming acoustic models: codec LMs, flow matching, delayed streams

What you will learn

• For each modern TTS design, what has to exist before the first audio sample can be produced.
• How CosyVoice 2, the reference streaming design, combines a token language model and chunked flow matching, and how its released code differs from its paper.
• How the frame-by-frame designs (Moshi, Sesame CSM, Qwen3-TTS, Kyutai's delayed streams) compare with a flow-matching second stage.
• What sentence-level streaming does for the CPU track's Kokoro, and what the GPU track's Qwen3-TTS achieves through vLLM-Omni.

Before you start

• Chapters: 3 (TTFA), 10 (pipeline, output and input streaming), 11 (codec frames, tokens per second).
• Tools: the ssh-tts-cpu environment for the CPU track; ssh-tts-gpu and the local GPU for the GPU track (code/tts/README.md).

The question for every design

Time to first audio (TTFA) is set by one question: what must the system have computed before it can produce sample 1? Speed matters only after that question is answered. A design that needs the whole utterance planned first has a TTFA proportional to the utterance's length, however fast it runs; a design that needs one frame has a TTFA of one frame's computation plus its look-ahead.

Codec language models

VALL-E writes EnCodec tokens with two models. An autoregressive model writes the first codebook frame by frame; a non-autoregressive model then fills in codebooks 2 to 8, attending to all positions, and "the NAR decoder will be called seven times" (Wang et al., arXiv:2301.02111, Sec. 4.2). Because the second model reads the whole first-codebook sequence, VALL-E as published produces no audio until the first model has finished the utterance.

XTTS streams by recomputation. Every 20 language-model tokens it decodes all the model's latents from the start with its HiFi-GAN decoder, holds back the last 1,024 samples of each decode and crossfades them into the next one (stream_chunk_size=20, overlap_wav_len=1024, TTS/tts/models/xtts.py, lines 618-693). Each chunk costs more than the last, because the decoder re-reads everything.

CosyVoice 2: the reference streaming design

CosyVoice 2 (Du et al., arXiv:2412.10117) is the design most streaming systems now resemble. It has three stages: a language model writes 25 Hz semantic tokens (chapter 11), a flow-matching model turns them into 50 Hz mel frames, and a HiFT vocoder (a neural source-filter vocoder with an inverse STFT, chapter 13) turns those into 24 kHz audio.

Bi-streaming in the language model

For input streaming, the language model is trained on text and speech tokens interleaved: "every N text tokens are followed by M speech tokens ... If the next token is a text token, the model is expected to predict a filling token ... N and M are set to 5 and 15" (Sec. 2.3). At inference (inference_bistream in cosyvoice/llm/llm.py, lines 552-640), the model reads 5 text tokens, writes 15 speech tokens, emits a filling token, and waits for 5 more text tokens ("not enough text token to decode, wait for more"). The filling token is not left to the model: the code inserts it on this fixed schedule, 15 speech tokens (600 ms of speech) per 5 text tokens. Text that needs more speech than that per token falls further behind on every round, the drift that arXiv:2603.06444 (chapter 10) names as a failure mode of input streaming.

Chunked flow matching

Flow matching refines all frames of its input together, so for output streaming CosyVoice 2 trains it with masks that let each frame see only its own chunk and the chunks before it. The paper describes four masks sampled uniformly: non-causal, full-causal, chunk-M and chunk-2M (p. 6). The released code trains with two, full context and a static 25-token chunk, each half of the time (streaming = True if random.random() < 0.5 else False, cosyvoice/flow/flow.py, lines 200-201). A look-ahead layer, "a right-padded 1-D convolution with the pad size of P and the kernel size of P + 1" with P = 3 (p. 5 and Fig. 3; pre_lookahead_len: 3 in the released configuration), lets each chunk see 3 tokens, 120 ms, past its end.

What the code waits for

The paper gives its "first-package latency" as a formula, L_TTS = M·d_lm + M·d_fm + M·d_voc, with M = 15 tokens (Sec. 2.5 and Table 8): the time for the language model to write M tokens, plus flow matching and vocoding for them. The released code waits for more (cosyvoice/cli/model.py, lines 257-361):

• the first chunk is self.token_hop_len = 25 tokens, not 15;
• plus prompt_token_pad, 0 to 24 tokens that align the voice prompt and the first chunk to the 25-token grid the model was trained on;
• plus the 3 look-ahead tokens;
• checked every 100 ms (while True: time.sleep(0.1)), so up to 100 ms more;
• later chunks grow: self.token_hop_len = min(self.token_max_hop_len, self.token_hop_len * self.stream_scale_factor), 25, then 50, then 100 tokens.

Worked out for the first request with no padding: the language model must write 25 + 3 = 28 tokens, 1.12 s of speech, before the first chunk is processed; the first chunk yields 840 ms of audio (50 mel frames of 480 samples, minus the vocoder's held-back overlap). With 24 tokens of padding it must write 52 tokens first. The December 2024 code used 50-token hops (2 * self.flow.input_frame_rate); the NVIDIA Triton runtime in the same repository uses 15 (runtime/triton_trtllm/.../model.py, line 81). Which "CosyVoice 2" you measure decides the first chunk.

One more detail matters for anyone measuring it. The grown hop is stored on the model object and never reset, so after one request with several chunks, token_hop_len stays at 100. An open upstream pull request, QwenAudio/CosyVoice#1849 (the repository has moved from FunAudioLLM/CosyVoice to QwenAudio/CosyVoice), describes the shared state; reading the code, a second request on the same object would start with a 100-token first chunk, four seconds of audio. Measure TTFA on a fresh model object, or reset the attribute per request, and run two requests back to back to check.

Each chunk also re-runs flow matching over the prompt and all tokens so far and keeps only the new frames (tts_mel[:, :, token_offset * self.flow.token_mel_ratio:]); fixed random noise per frame index and the causal mask make the earlier frames come out the same. The cost of chunk k therefore grows with its position.

What does streaming cost in quality? On the hard test set of Seed-TTS-eval (the common intelligibility benchmark described in chapter 14; its hard set is about 400 Mandarin sentences with repetitions and tongue twisters), CosyVoice 2's error rate goes from 6.83 offline to 8.08 with both stages streaming; streaming only the flow-matching stage gives 7.12, streaming only the language model 7.88 (Table 8). "The primary impact of the streaming LM is observed in challenging cases from the test-hard set, likely due to the loss of contextual information in streaming mode" (p. 12). On ordinary sentences the difference is small.

The "150 ms"

CosyVoice's README says it "achieves latency as low as 150ms". The line was added on 2024-12-16, the day before the first CosyVoice 2 code and now sits under Fun-CosyVoice 3. Neither paper contains a measured 150 ms. The measurements that do exist are in chapter 14: an average first-chunk TTFA of 220 ms at one request on one L20 card through the repository's Triton and TensorRT-LLM runtime, and 1,643 ms to the first packet with text fed incrementally, measured by another team on an A100.

CosyVoice 3 keeps the same streaming schedule, replaces the flow model by a 22-layer DiT (depth: 22 in the released cosyvoice3.yaml) (a diffusion transformer, a transformer used as the flow-matching network), and uses a causal HiFT vocoder that re-vocodes all mel frames so far and slices off what was already sent (cosyvoice/cli/model.py, lines 397-450; arXiv:2505.17589).

Flow matching or frame by frame

The second stage can be a chunked flow-matching model, as in CosyVoice, or a model that produces the acoustic codes frame by frame. FireRedTTS-1S implemented both and measured them on the same NVIDIA L20: chunk-wise flow matching with 1 s chunks gives "the RTF of 0.1 and latency < 300ms"; a multi-stream acoustic language model gives "a higher RTF of 0.3 but a latency < 150ms" (arXiv:2503.20499, pp. 4-5). The flow path's latency bound, (l + 3)·t_s + t_a + t_c, has the same 3-token look-ahead as CosyVoice 2.

This is the clearest published example of chapter 3's rule that TTFA is set by a chunk policy, not by speed: the design that is three times slower overall has half the bound on its TTFA.

Backbone plus depth transformer

Chapter 11 showed that writing every code of every frame as a separate token is too slow at 12.5 Hz with 16 or 32 codebooks. The answer is two transformers. A large backbone (or temporal transformer) takes one step per 80 ms frame; a small depth transformer then writes the codes of that frame one by one.

• Moshi: a 7B temporal transformer (dimension 4,096, 32 layers) and a depth transformer (dimension 1,024, 6 layers) over Mimi's codebooks (kyutai-labs/moshi loaders.py, lines 87-113). Its acoustic codebooks lag the semantic one by one frame, for "a theoretical latency of 160ms (80ms for the frame size of Mimi + 80ms of acoustic delay), with a practical overall latency as low as 200ms on an L4 GPU" (moshi README).
• Sesame CSM: the backbone predicts codebook 0 of each frame, a small decoder the other 31. The open CSM-1B code is not streaming: it generates all frames, then decodes Mimi once and watermarks the whole clip (SesameAILabs/csm generator.py, lines 146-165).
• Qwen3-TTS (the GPU track): the backbone predicts codebook 0 of the 12.5 Hz tokenizer, and "an MTP (Multi-Token Prediction) module then generates all residual codebooks" (Hu et al., arXiv:2601.15621, p. 4). It streams in packets: "we define one speech packet as 4 tokens, which means 320 ms of speech per packet" (p. 6), decoded without look-ahead.

Qwen3-TTS's paper reports a time to first audio ("first-packet latency") of 97 ms for the 0.6B 12 Hz model and 101 ms for the 1.7B one, at one request (Table 2), measured "on our internal vLLM engine (vLLM V0 backend) on a single typical computational resource" (Sec. 3.4), a device it does not name. At three concurrent requests the 1.7B model's figure is 195 ms; at six, 333 ms. The open repository's own inference code does not stream: its non_streaming_mode=False "only simulates streaming text input ... rather than enabling true streaming input or streaming generation" (qwen3_tts_model.py, lines 513-515). The GPU track therefore serves it with vLLM-Omni, measured below and in chapter 16.

Delayed streams

Kyutai's TTS (Delayed Streams Modeling, Zeghidour et al., arXiv:2509.08753) treats text and audio as two streams of the same 12.5 Hz clock, with the audio shifted behind the text. The released checkpoint, kyutai/tts-1.6b-en_fr, says it plainly: "The audio is shifted by 16 steps (1.28 sec.) with respect to the text, and the model uses an acoustic/semantic delay of 2"; its configuration has "audio_delay": 1.28 and a second text stream ahead of the first ("second_stream_ahead": 2; the configuration gives no unit) (model card and config.json, revision f654396). It is 1.8B parameters in total, a 1B backbone and a depth transformer, over 32 Mimi codebooks; the weights are CC-BY 4.0.

During the first 16 steps there is no audio to write, and the code skips the depth transformer ("we don't need to run the depformer since the audio stream should be all token_ids.zero", moshi/models/tts.py, lines 614-617). The paper's formula for the TTFA is (τ·f_r − 1)·d_b + 3·(d_b + d_q), where τ is the delay in seconds and f_r the frame rate (12.5 Hz), with d_b = 5 ms per backbone step and d_q = 25 ms per depth pass: for the shipped 16 steps, 15 × 5 + 3 × 30 = 165 ms. The paper reports 150 ms at batch size 1 on an H100 (Table 6), and states the delay three different ways (1.28 s, "1.6s, or 16 steps", and 2 s in an appendix), none of which gives 150 ms from its own formula. Read 150 ms as measured, 165 ms as the mechanism. Its "RTF" is the inverse of everyone else's: audio time per compute time, higher is better (3.2 at batch 1).

The streaming script decodes Mimi frame by frame as frames arrive; the plain tts_pytorch.py script "waits for all the text to be available before starting the audio generation" (delayed-streams-modeling README).

Measured: the CPU track, Kokoro

Kokoro predicts all durations of its input at once (chapter 10), so its only streaming is by text segment. I measured three ways of synthesizing 5 three-sentence paragraphs per language, each run 3 times (15 runs per cell), on four threads of an Intel i5-12600K (code/tts/ch12_kokoro_cpu/kokoro_stream.py; code/tts/results/ch12_kokoro_cpu.json). TTFA is the arrival of the piece of audio that contains the first audible sample (the first 10 ms window at or above −40 dBFS). Kokoro's output begins with about 0.3 s of near-silence in both configurations measured here (0.28 to 0.36 s), which the player plays, so a listener hears speech about 0.3 s after the TTFA below; a player could trim it.

• whole: the paragraph in one call;
• sentence: all text known, one call per sentence, each sentence's audio handed over as soon as it exists;
• stream: the text arrives at 20 words (or characters) per second, like tokens from a language model, and a sentence is synthesized once its last word has arrived.

Language, mode	TTFA p50	TTFA p95	RTF p50	Runs that underran (of 15)
English, whole	4,024 ms	5,386 ms	0.21	0
English, sentence	1,137 ms	2,893 ms	0.19	3
English, stream	1,919 ms	4,816 ms	0.24	3
Mandarin (v1.1-zh), whole	1,922 ms	2,441 ms	0.19	0
Mandarin (v1.1-zh), sentence	907 ms	1,048 ms	0.18	0
Mandarin (v1.1-zh), stream	1,742 ms	2,077 ms	0.24	0

Splitting into sentences cut the English median TTFA by a factor of three and a half, with no change to the model. The p95 stays high because one of the five English paragraphs opens with a 40-word sentence (13.9 s of audio), whose three runs took 2.73, 2.85 and 3.01 s to first audio on identical input: on single sentences, Kokoro's TTFA grew by about 0.07 s per English word. Streaming the text in adds the time the first sentence takes to arrive (about 0.8 s at 20 words per second for these paragraphs). And one English paragraph underran in all three of its runs, in both sentence modes: the player ran out of audio. Its first sentence, "Why did Lord Arthur Fluffinose enter?", is 2.8 s of audio and took 0.52 s to synthesize; its second is 14.9 s of audio and took 3.12 s. Playback of the first sentence began at 0.52 s and ended at 3.29 s, but the second was ready only at 3.64 s: 0.34 s of silence in the middle of the paragraph (0.34, 0.34 and 0.19 s in the three runs), although every sentence was synthesized about five times faster than real time. Chapter 14 is about exactly this.

Measured: the GPU track, Qwen3-TTS on vLLM-Omni

I served Qwen3-TTS (Qwen3-TTS-12Hz-1.7B-CustomVoice) with vLLM-Omni 0.32.0 on one RTX 4090, with vLLM-Omni's default deploy file (deploy/qwen3_tts.yaml), and sent it the 5 three-sentence paragraphs per language of the Kokoro measurement, each twice, one request at a time (code/tts/gpu/qwen3tts_client.py, input-modes; code/tts/results/ch12_qwen3tts_input_modes.json). I ran seven conditions; the table shows four:

• all text at once over HTTP (the WebSocket with the whole text in one message, in any server mode, gave the same TTFA, 299 to 303 ms at the median);
• streamed in at 20 words or characters per second over the WebSocket, as from a language model, with the server's split_granularity off (synthesize at the end), at sentence, or in commitment mode (chapter 16).

Text, server mode	TTFA p50 / p95, English	TTFA p50 / p95, Mandarin	RTF p50 (en)
all at once, HTTP	305 / 314 ms	303 / 309 ms	0.13
streamed in, no split	3,256 / 3,707 ms	2,410 / 3,057 ms	0.25
streamed in, sentence split	1,092 / 2,252 ms	1,197 / 1,403 ms	0.19
streamed in, commitment	1,095 / 2,257 ms	1,193 / 1,407 ms	0.20

Three readings.

1. The first packet is not the first audio. With all text at once, the first piece of PCM arrived after about 36 ms, but it was the server's 80 ms first chunk (1,920 samples) and it was silent. The first audible sample came in the next piece, at about 305 ms. A client that counts the first byte, as a "first packet" number does, would report 36 ms for what the listener hears at 305 ms.
2. Splitting is what makes input streaming pay. Streamed in without splitting, the server waited for the whole text (the last word arrived at 2.95 s in English) and the first audio came 0.31 s after that. With sentence splitting or commitment mode the first sentence was synthesized as soon as it was complete (at 0.80 s in English, 0.85 s in Mandarin), and the first audio came about 0.30 s later in English and 0.35 s in Mandarin. The English p95 of 2.25 s comes from paragraphs whose first sentence is long, as with Kokoro.
3. The model's own pace is fast enough. Once audio flows it arrives in 2 s pieces, about every 260 ms in English (the 25-frame chunks of chapter 16), well ahead of playback: no request underran, at a median RTF of 0.13 to 0.33 (for the streamed-in modes the RTF counts from the first word of text, so it includes the wait for the text).

So on this card the GPU track reaches first audio about 0.30 s after the first sentence of text is complete in English and 0.35 s in Mandarin. With all text at once it takes about 305 ms. The paper reports 101 ms for the same model size at one request, about 205 ms less; this measures more (a client over HTTP, to the first audible sample) on a different engine and a different card.

Common misconceptions

"CosyVoice 2 streams in 15-token chunks." The paper does; the released Python code waits for 25 tokens plus padding plus 3 look-ahead tokens, and later chunks grow to 100.

"Faster model, sooner audio." FireRedTTS-1S's slower decoder has half the TTFA bound of its faster one.

"A delayed-streams model must wait 1.28 s." It waits 16 steps, but steps without audio are cheap: the depth transformer is skipped, and the formula gives 165 ms.

Try it

CPU track

1. python ch12_kokoro_cpu/kokoro_stream.py --threads 4 from code/tts/ reproduces the table above. Change --threads to 1 and see which column changes and which does not.
2. Edit one test paragraph so it starts with a very short sentence followed by a very long one, and run it in sentence mode. Predict whether it underruns before you run it.

GPU track

code/tts/gpu/run_all_gpu.sh starts vLLM-Omni with Qwen3-TTS on the local GPU and runs the input-mode measurement. Compare its TTFA with the paper's 101 ms at one request, and note everything that differs: engine, card, model variant, what is measured.

English and Mandarin

Kokoro's Mandarin TTFA in sentence mode had a p95 of 1,048 ms against 2,893 ms in English, because the Mandarin test sentences are shorter. Compare TTFA per unit of text instead: the slope was about 0.035 s per character in Mandarin and 0.07 s per word in English.

Key terms

• codec language model: a language model that writes codec tokens as its output.
• interleaving (N:M): alternating N text tokens and M speech tokens in one sequence, for input streaming.
• filling token: CosyVoice 2's token meaning "give me the next text tokens".
• chunk mask: an attention mask that lets a frame see only its own chunk and earlier ones.
• backbone / depth transformer: one large model step per frame, and a small model for the codes inside it.
• packet: the unit of audio a streaming TTS hands over; 4 tokens, 320 ms, for Qwen3-TTS 12 Hz.
• delayed streams: text and audio on one clock, the audio shifted a fixed number of steps behind.
• underrun: the player runs out of audio before the next chunk arrives.

Sources

• Wang et al., "Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers" (VALL-E), arXiv:2301.02111.
• coqui-ai/TTS (commit dbf1a08, branch dev), TTS/tts/models/xtts.py.
• Du et al., "CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models", arXiv:2412.10117; Du et al., "CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training", arXiv:2505.17589; FunAudioLLM/CosyVoice (commit 074ca6d), cosyvoice/cli/model.py, cosyvoice/llm/llm.py, cosyvoice/flow/flow.py, README.md; pull request QwenAudio/CosyVoice#1849.
• "FireRedTTS-1S: An Upgraded Streamable Foundation Text-to-Speech System", arXiv:2503.20499.
• Défossez et al., "Moshi: a speech-text foundation model for real-time dialogue", arXiv:2410.00037; kyutai-labs/moshi (commit e6a55d2).
• SesameAILabs/csm (commit daed31e), generator.py, models.py.
• Hu et al., "Qwen3-TTS Technical Report", arXiv:2601.15621; QwenLM/Qwen3-TTS (commit 022e286).
• Zeghidour et al., "Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling", arXiv:2509.08753; kyutai-labs/delayed-streams-modeling (commit 4c4f65e); model card kyutai/tts-1.6b-en_fr (revision f654396).
• hexgrad/kokoro (commit dfb907a); model cards hexgrad/Kokoro-82M (revision f3ff357) and hexgrad/Kokoro-82M-v1.1-zh (revision 01e7505).