16. Serving streaming TTS

What you will learn

• How a streaming TTS server is built from two stages, a token-generating language model and a codec decoder, and what passes between them, with vLLM-Omni serving Qwen3-TTS as the worked example.
• What the server's chunk settings do to time to first audio and to the risk of underruns.
• The HTTP and WebSocket interfaces of a streaming speech API, and what text arriving word by word changes.
• How TTFA, RTF and underruns move with concurrency, where the capacity knee is, and how to find it on your own card.

Before you start

• Chapters: 11 (codec frames), 12 (Qwen3-TTS, packets), 13 (causal decoders keep state), 14 (TTFA, deadlines, underruns), 15 (stream state and batching).
• Tools: the ssh-tts-gpu environment and the local GPU (code/tts/README.md); the load generator in code/serve/ch16/.

Two stages

A codec-language-model TTS has two very different kinds of work. The talker, a language model, writes codec frames one step at a time, keeps a KV cache per request, and benefits from the same continuous batching as any LLM server. The code2wav stage, the codec decoder, turns frames into samples; it is a convolutional and attention network that keeps a decoder state per request (chapter 13) and runs on chunks of frames.

vLLM-Omni serves Qwen3-TTS exactly this way. Its deploy file defines stage 0, the talker, and stage 1, code2wav, joined by a shared-memory connector, with streaming enabled by async_chunk: true (vllm_omni/deploy/qwen3_tts.yaml at v0.32.0). The file's header says it was "Verified on 1x H100".

What passes between them

The talker hands frames to code2wav in chunks. A chunk is released only when enough frames have accumulated (if not finished and adjusted % chunk_size != 0: return None), and after the first chunk only the new frames travel, because "Code2Wav keeps the quantizer/conv/Transformer context per request" (stage_input_processors/qwen3_tts.py). The deploy file sets the sizes:

• initial_codec_chunk_frames: 1: "Emit only the first audio chunk early, then return to codec_chunk_frames";
• codec_chunk_frames: 25: "Must match the decoder sliding attention window";
• codec_left_context_frames: 72;
• max_num_seqs: 64 on both stages, and batches of at most 4 chunks in the decoder.

In time, with 80 ms frames: the first chunk is 80 ms of audio, every later chunk 2,000 ms, and the decoder keeps 5.76 s of left context. Stage 1 receives 16 codes per frame, 200 codes per second of audio per stream.

The cliff after the first chunk

Chapter 14's deadline rule applies directly. After the 80 ms first chunk, the player holds 80 ms of audio. The second chunk is 25 frames; for the player not to run dry, it must arrive within 80 ms of the first, or within 180 ms if, like vLLM-Omni's own benchmark, you tolerate a 100 ms gap. Producing 25 frames in 180 ms is about 139 frames per second for that one stream, eleven times the real-time rate of 12.5 frames per second. A loaded server cannot do that. The deploy file therefore has an optional ramp, codec_chunk_ramp (example [4, 4, 8, 16, 25]), "to eliminate the IC→steady cliff (1→25)": chunks grow gradually, so each deadline is reachable.

For the GPU track's model the cliff is hidden by what the first chunk contains. In my measurements (chapter 12) the 80 ms first chunk arrived about 36 ms after the request and was silent: every utterance starts with near- silence. The first audible sample came with the second piece, usually a 2 s chunk, at about 305 ms. So the listener's first audio is the 25-frame chunk, not the 1-frame one, and a player that starts on the first audible sample holds 2 s of audio from the start; the early chunk only tells the client the stream is alive. A time-to-first- packet number for this server measures that silent chunk.

The documentation also warns against the obvious latency trick of decoding one stream at a time: "stage 1 performed best at max_num_seqs: 10", and "avoid forcing stage 1 back to max_num_seqs: 1" (docs/serving/speech_api.md, lines 902-913). The first is the throughput advice, measured on the CustomVoice checkpoint this book uses; the second is the docs' advice for latency-sensitive deployments. The deploy file's own comment on stage 1 says the opposite for the same checkpoint: "Tuned for Base voice clone; CustomVoice / VoiceDesign are TTFA-optimal at 1" (vllm_omni/deploy/qwen3_tts.yaml, line 119). The book's runs used the file's default, 64.

For CosyVoice 3, served by the same project in two stages (talker plus flow-matching code2wav), the deploy file uses 15-token chunks with 25 tokens of left context and adds a warning worth quoting: "Streaming continuity is GPU-class sensitive — TUNE PER GPU". On an H20-3e the same configuration "regressed continuity to ~78% at c=4 (vs ~86% with smaller batch/chunk), because code2wav can't produce mel chunks fast enough ... → buffer underruns → audible gaps" (vllm_omni/deploy/cosyvoice3.yaml). With 15 tokens of 40 ms plus 3 look-ahead tokens, 720 ms of speech tokens must exist before its first decode.

The interface

vLLM-Omni exposes three speech routes: POST /v1/audio/speech, POST /v1/audio/speech/batch, and a WebSocket at /v1/audio/speech/stream (vllm_omni/entrypoints/openai/api_server.py). Three details trip up first-time clients:

• stream=true alone returns server-sent events (speech.audio.delta, speech.audio.done), not audio bytes; raw PCM needs stream_format="audio" as well. This became the default in v0.24.0.
• Every streaming mode requires response_format pcm or wav.
• Headerless PCM cannot say its own sample rate, so the server sends it in headers, X-Audio-Sample-Rate and X-Audio-Channels (serving_speech.py, lines 89-90). A client that assumes 24 kHz for a 22.05 kHz model plays it at the wrong speed; vLLM-Omni's benchmark README notes the same trap in its own harness, where a 44.1 kHz model would report an audio duration 1.84 times too long and an RTF that "looks 1.84x better than reality".

The WebSocket protocol is the one a voice agent uses. The client sends session.config, then input.text messages as text arrives, then input.done, which "is a flush, not a close": the next utterance reuses the connection. The server answers with audio.start, binary audio frames, audio.done (serving_speech_stream.py, lines 4-56). How text that arrives in pieces becomes synthesis is set by text_input_mode. In buffered mode (the default) the server keeps all the text and synthesizes once at input.done, unless split_granularity is sentence or clause. In commitment mode it starts an independent request at each sentence end; the documentation is candid that "each segment is an independent request ... so seamless cross-segment prosody is not guaranteed". Chapter 18 measures what this does inside a voice agent.

Concurrency and the capacity knee

Streaming moves work around rather than saving it. vLLM-Omni's design notes measured Qwen3-TTS on an H200 with and without asynchronous chunking: at one request, the time to first packet fell from 733 to 64 ms while the RTF rose from 0.124 to 0.160; in their words, "Async chunk trades ~30% higher total compute for 11x faster time-to-first-audio" (docs/design/qwen3_omni_tts_performance_optimization.md). That table predates the current left-context setting.

As concurrency rises, the first sign of trouble is not RTF. vLLM-Omni's benchmark README prints this, "Observed on H20-3e, Qwen3-TTS-1.7B (measured pre-merge on this branch)":

Concurrency	1	4	8	16	32
1.7B-Base, RTF	0.15	0.28	0.49	0.72	0.77
1.7B-Base, time to first packet	165 ms	412 ms	1,701 ms	3,355 ms	3,772 ms

RTF stays below 1 all the way to 32 streams. Time to first packet quadruples between 4 and 8. A capacity limit defined by RTF would put this server's knee above 32; one defined by first-packet time puts it between 4 and 8. The book uses the second kind: the capacity knee is the lowest concurrency at which TTFA p95 leaves its target, or at which the share of requests that play without a gap falls below its target. (The README attributes the jump to an issue number that, at this commit, points to an unrelated model request; the table itself is what was measured.)

Baseten's numbers for Orpheus, an LLM that writes SNAC codes, show the same arithmetic from the other side: "For Orpheus, 83 tokens is roughly equivalent to one second of audio", and "Our original implementation of Orpheus supported 7 simultaneous real-time streams on an H100 MIG GPU ... After optimizing the model with TensorRT-LLM, we see a base rate of 16 simultaneous streams or up to 24 for applications with stable traffic patterns", with "a 200-millisecond TTFB on H100 MIG" (Baseten blog, 2025-05-05). Sixteen streams at 83 tokens per second each is 1,328 tokens per second of aggregate generation: a TTS server's capacity is its token throughput divided by the per-stream real-time rate, as long as the first-audio deadline is also met. Baseten's guide adds that "streaming over WebSockets is the biggest unlock for performance versus sending text and receiving audio in discrete chunks" (Inference Engineering, Sec. 6.4).

How vLLM-Omni measures it

The project's benchmark, vllm bench serve --omni with --backend openai-audio-speech, measures time to first packet on the client, from before the HTTP post to the first non-empty response chunk (vllm_omni/benchmarks/patch/patch.py); RTF as total request latency over audio duration; and audio continuity with a simulated player that starts at the first chunk with no pre-buffer: at each later arrival, the deficit is the audio the player should have played minus the audio received, and a request is continuous if its largest deficit is at most 0.1 s (vllm_omni/benchmarks/audio_continuity.py). Its docstring names the reason: it "Captures the failure mode where RTF_p50 < 1 ... but per-stream chunk arrival is bursty enough that listeners hear audible gaps." One caveat: its "chunks" are the client's socket reads, not the server's chunks. The book's harness (chapter 14) differs in two places. Its TTFA stops at the first audible sample, not the first non-empty chunk, and its player starts there too; it pauses when it runs dry, resumes when audio arrives, and counts underruns and total stall time, while vLLM-Omni's clock never pauses. On Qwen3-TTS the two give opposite verdicts. Its first chunk is 80 ms of silence and the next arrives about 0.27 s later, so vLLM-Omni's player has run dry before any speech arrives. Applied to the saved arrival times, its rule marks 0 of 712 requests from chapter 14's load test as continuous (the smallest worst deficit is 0.167 s), at every load, while the book's player found no gap in any request up to 64 concurrent. Before comparing continuity numbers, check where each tool starts its player.

Measured: the GPU track under load

The book's load generator (code/serve/ch16/tts_openloop.py) sends requests to the GPU track's vLLM-Omni server in two ways. In the open loop, requests arrive as a Poisson process at a fixed rate and are sent whatever is still in flight, as real users do; chapter 14's closed loop, where each worker waits for its previous request, never lets more than N requests pile up. In the streamed-text load, N WebSocket sessions each receive their text word by word at 20 units per second, as from a language model, with split_granularity=sentence: the path a voice agent uses.

Open loop, one sentence per request over HTTP, the project's default deploy file (code/serve/results/ch16_openloop.json and ch16_openloop_high.json):

Arrivals per second	Audio asked for per second	Requests	TTFA p50 / p95	RTF p50	In flight at arrival, p95	Requests that underran
0.5	3 s	50	310 / 349 ms	0.14	1.0	0
2	12 s	50	337 / 397 ms	0.16	4.0	0
4	25 s	50	359 / 401 ms	0.17	5.5	0
6	37 s	50	402 / 510 ms	0.19	12.5	0
8	50 s	50	411 / 476 ms	0.19	10.0	0
10	58 s	80	510 / 647 ms	0.24	19.0	0
12	71 s	80	549 / 620 ms	0.26	19.0	0
16 (overload)	99 s	80	733 / 1,110 ms	0.35	41	0
20 (overload)	121 s	80	795 / 1,082 ms	0.38	45	0

Read the second column first. It is the arrival rate times the mean audio per request (about 6 s): the seconds of speech the server is asked to produce per second. Chapter 14 measured how much audio per second this server makes; at 64 workers it made 89.3 s per second. The last two rows ask for more than that, so there the server is asked for more than it can make. An open loop past capacity has no steady state: the queue grows for as long as requests keep coming. Those two rows are short bursts of 80 requests, sent in 5 and 4 s, and their numbers describe how far the queue grew before the burst ended, not a level the server would hold. No request underran even there.

Held to a TTFA p95 of 1 s, this server's knee is between 12 and 16 arrivals per second (620 against 1,110 ms). The p95 of the next rate, 1,082 ms at 20, is a little below the one at 16: with 80 requests the p95 is set by the four slowest, so it is a band, not a point. No request played with a gap at any rate, so the gap-free criterion gives no knee in this range. The TTFA target is the only one that is crossed, and it is crossed where the audio asked for passes what the server can make.

With the text streamed in at 20 units per second over the WebSocket (split_granularity=sentence, N sessions, 40 sentences per level; code/serve/results/ch16_wsload.json), TTFA counted from the first word of text was 1.00 s at the median with one session, 1.05 s with 8 and 1.14 s with 16, and nothing underran. About 0.7 s of that is the text arriving: the last unit of the median test sentence was sent 0.78 s after the first in English and 0.68 s in Mandarin. The rest, from the last unit of text to the first audible sample, is the server's share: a median 0.30 s with one session, 0.38 s with 8 and 0.46 s with 16.

Common misconceptions

"RTF below 1 means the server has capacity left." On the H20-3e table, RTF was 0.49 at 8 streams while the first packet took 1.7 s.

"A one-frame first chunk gives low latency for free." It moves the problem to the second chunk's deadline, which a loaded server misses without a ramp.

"stream=true returns audio." In vLLM-Omni it returns events; ask for stream_format="audio".

Try it

GPU track

1. Start the server with code/tts/gpu/with_server.sh (it waits for /health, runs one client command, and stops the server) and request one sentence with stream=true, then with stream_format="audio". Look at the difference in what arrives, and read X-Audio-Sample-Rate.
2. Run code/serve/ch16/tts_openloop.py at increasing rates (see its docstring) and find the knee by both definitions. Which one comes first? Compute the audio asked for per second at each rate, and stop at the server's capacity: past it, longer runs give longer queues, not a new point.
3. Enable codec_chunk_ramp in a copy of the deploy file, pass it to the server with DEPLOY_CONFIG=<copy> code/tts/gpu/with_server.sh ..., and rerun the point just past the knee. Compare the underruns and stall time the harness reports.

CPU track

The CPU track's Kokoro has no streaming server in this book. Write the smallest one that would work: a WebSocket endpoint that receives text, splits it into sentences (chapter 10), synthesizes them in order on one CPU thread pool, and sends PCM as each sentence finishes. Then apply chapter 14's deadline rule to predict, from the sentence-level TTFA slope of chapter 12, which paragraphs it would underrun on.

English and Mandarin

commitment mode splits at sentence terminators for Chinese and English (. ! ? 。 ！ ？ …). Send a long Mandarin sentence with only commas through the WebSocket in both modes and compare when the first audio arrives.

Key terms

• talker / code2wav: the token-generating stage and the codec-decoding stage of a two-stage TTS server.
• async chunking: handing codec frames to the decoder in chunks while the talker is still writing.
• chunk ramp: chunk sizes that grow gradually after the first, so that each chunk's deadline can be met.
• audio continuity: whether a request plays without a gap longer than a threshold (0.1 s in vLLM-Omni).
• open-loop load: requests sent at a fixed rate regardless of the server's progress.
• capacity knee: here, the lowest load at which TTFA p95 or continuity leaves its target.

Sources

• vllm-project/vllm-omni v0.32.0 (commit 5917730): vllm_omni/deploy/qwen3_tts.yaml, vllm_omni/deploy/cosyvoice3.yaml, vllm_omni/model_executor/stage_input_processors/qwen3_tts.py, vllm_omni/entrypoints/openai/api_server.py, vllm_omni/entrypoints/openai/serving_speech.py, vllm_omni/entrypoints/openai/serving_speech_stream.py, vllm_omni/entrypoints/openai/protocol/audio.py, vllm_omni/benchmarks/audio_continuity.py, vllm_omni/benchmarks/patch/patch.py, benchmarks/tts/README.md, docs/serving/speech_api.md, docs/design/qwen3_omni_tts_performance_optimization.md; release notes of v0.24.0.
• Hu et al., "Qwen3-TTS Technical Report", arXiv:2601.15621.
• canopyai/Orpheus-TTS (commit e64661f); hubertsiuzdak/snac_24khz configuration (revision d73ad17).
• Baseten, "Canopy Labs selects Baseten as preferred inference provider for Orpheus TTS model" (blog, 2025-05-05, updated 2025-05-20), and Inference Engineering, Sec. 6.4, fetched 2026-10-29.