14. Measuring streaming TTS

What you will learn

• What each published "first-packet latency" actually measures, and how to measure time to first audio yourself on the client.
• Why real-time factor is not enough (chunk deadlines and underruns), and how TTFA moves as concurrency grows.
• How intelligibility, speaker similarity and naturalness are scored, what the common Seed-TTS-eval scripts actually compute, and what they skip.
• What streaming costs in quality compared with offline synthesis of the same text.

Before you start

• Chapters: 3 (TTFA, percentiles, concurrency), 9 (WER, CER, normalizers), 12 (the TTS systems measured here).
• Tools: the ssh-tts-cpu environment for the CPU track; ssh-tts-gpu and ssh-tts-eval with a GPU for the GPU track and the intelligibility scoring (code/tts/README.md).

Time to first audio: whose definition?

Every paper and repository reports a first-audio number, and almost no two measure the same span:

Source	Name	Starts at	Ends at	Leaves out
CosyVoice 2 paper	"first-package latency"	—	—	everything but model compute: it is a formula, M·d_lm + M·d_fm + M·d_voc
Qwen3-TTS paper	"first-packet latency"	request in the engine	first packet decoded	text arrival, network, client; internal vLLM V0 engine, unnamed device
Kyutai DSM paper	time to first audio	request	first audio, "including all computations"	network; one H100
VoXtream paper	"first-packet latency"	first text	first packet	network; text fed incrementally ("full-stream"); A100, FP16
Moshi paper	theoretical latency	—	—	everything but frame size plus acoustic delay (160 ms); "200 ms in practice" on an L4
CosyVoice Triton client	first-chunk latency	client, just before the gRPC call	client, first response callback	playback; one L20

Sources: arXiv:2412.10117 Sec. 2.5; arXiv:2601.15621 Sec. 3.4; arXiv:2509.08753 Table 6 and p. 22; arXiv:2509.15969 Table 3; arXiv:2410.00037 and the moshi README (commit e6a55d2, lines 39-40, for "on an L4 GPU"); CosyVoice runtime/triton_trtllm/client_grpc.py lines 70-89 and 424-433. Only the last is measured on the client, where the user is.

The book uses one definition, measured by the client (code/tts/hbtts/stream.py): TTFA is the time from the moment the client sends the request (or the first piece of text, when text streams in) to the arrival of the piece of audio that contains the first audible sample. "Audible" is the first 10 ms window at or above −40 dBFS (chapter 1): many systems send a WAV header or a few hundred milliseconds of near-silence first, and neither is something a user hears. The harness also records the arrival of the first PCM byte, as a diagnostic, because the gap between the two is a measurement in itself.

Real-time factor, and which way round

RTF in TTS is generation time divided by the duration of the audio generated. Below 1, the system produces audio faster than it plays. But the sources do not agree on the direction. F5-TTS, FireRedTTS-1S, Qwen3-TTS and CosyVoice's logs report compute time over audio time, lower is better. VoXtream's paper does both: its text defines RTF as "the ratio of generated speech duration to wall-clock generation time", while its table is headed "RTF ↓" and its values (0.17 to 1.00) behave as compute over audio (arXiv:2509.15969, Secs. 3 and 4). Kyutai's DSM paper reports the inverse: "Real-Time Factor (RTF) is higher than 1 if the model can produce audio in real time" (arXiv:2509.08753, Table 6), and vocoder papers speak of "times faster than real-time". Check the direction before comparing.

CosyVoice's own log line computes something else again: for each chunk, the time since the previous yield divided by that chunk's audio duration, starting after the text front end (cosyvoice/cli/cosyvoice.py, lines 97-103). So the first chunk's logged RTF excludes normalization and prompt processing.

Chunk deadlines and underruns

An RTF below 1 over the whole utterance does not mean the audio plays without a gap. Playback is a clock: once the first audio starts, the player needs one second of audio per second. Playback stays continuous if, and only if, every chunk arrives before the audio already received runs out: chunk k must arrive by TTFA plus the total duration of chunks 1 to k − 1. A chunk that misses its deadline causes an underrun, a silence in the middle of the speech.

The deadline is tightest early, when little audio has been received. Chapter 12's Kokoro measurement shows it on real data:

The first sentence, 2.8 s of audio, arrived at 0.52 s, so the second had to arrive by 3.29 s. It was 14.9 s of audio, took 3.12 s to synthesize, and arrived at 3.64 s: 0.34 s of silence. Its own RTF was 0.21; the paragraph's, 0.20. Every number that averages over the utterance says this paragraph was fine.

The same arithmetic applied to CosyVoice 2's code schedule from chapter 12 (840 ms of audio in the first chunk, then 2 s and 4 s per chunk, as the hop doubles from 25 to 50 to 100 tokens): the second chunk must arrive within 840 ms of the first, the third within 2.84 s. The second chunk has the tightest deadline.

So the client-side measurements that matter, all computed by the book's harness from the arrival time and size of every piece of audio, are:

• TTFA, as above;
• inter-arrival time between chunks, against the duration of the chunk before;
• underruns and the total stall time of a simulated real-time player that starts at the first audio;
• RTF last, as a summary.

A player can trade TTFA for safety with a pre-buffer: wait a fixed time after the first audio arrives before starting to play, so that more audio has accumulated by then. The harness simulates that too (prebuffer_s, a wait in seconds after the first audible piece arrives).

TTFA under load

A TTFA measured with one request says little about a server with many. The published measurements that state their concurrency all show the same pattern:

System, hardware	1 request	2 or 3	4 or 6	Source
CosyVoice 2, one L20, Triton and TensorRT-LLM, first chunk (average)	220 ms	477 ms (2)	1,107 ms (4)	runtime/triton_trtllm/README.Cosyvoice2.Unet.md
same, with a cached speaker embedding	190 ms	323 ms (2)	978 ms (4)	same
CosyVoice 3, one L20, 4 concurrent requests	—	—	750 ms average, 1,002 ms p99	README.Cosyvoice3.md
Qwen3-TTS 12 Hz 1.7B, internal engine, unnamed device	101 ms	195 ms (3)	333 ms (6)	arXiv:2601.15621, Table 2

For CosyVoice 2 on one L20, four times the load made the first chunk five times slower, while the RTF improved from 0.124 to 0.092: the server did more work per second by batching, and each request waited longer for its turn. Throughput and TTFA move in opposite directions. Report both, at a stated concurrency, with the percentile and the number of requests. Another team measured CosyVoice 2 with text fed incrementally on an A100 at 1,643 ms (VoXtream, arXiv:2509.15969, Table 3), a reminder that input streaming adds the wait for text.

Measured: the GPU track under load

I ran the GPU track's server, vLLM-Omni 0.32.0 with Qwen3-TTS (12 Hz, 1.7B CustomVoice), on one RTX 4090 against a closed loop of N clients: each sends a sentence-length request (the book's 20 English and 20 Mandarin sentences; at one request their audio ran from 2.1 to 13.0 s, median 4.6 s), waits until all of its audio has arrived, and sends the next. The player starts at the first audible piece. The RTF counts from the request, so it includes the wait for the first audio. Every level began with one untimed burst of N requests (code/tts/gpu/qwen3tts_client.py, concurrency; code/tts/results/ch14_qwen3tts_concurrency.json). The server ran with its default limit of 64 concurrent sequences (max_num_seqs, set in both pipeline stages), so at 64 clients every client is admitted at once.

Concurrent requests	Requests	TTFA p50 / p95	RTF p50 / p95	Requests that underran	Audio produced per second
1	40	306 / 310 ms	0.14 / 0.15	0	6.8 s
4	40	352 / 368 ms	0.16 / 0.18	0	23.4 s
8	40	426 / 459 ms	0.19 / 0.21	0	34.9 s
16	48	542 / 616 ms	0.24 / 0.27	0	51.4 s
32	96	834 / 944 ms	0.38 / 0.42	0	71.2 s
48	144	1,128 / 1,246 ms	0.50 / 0.57	0	82.7 s
64	192	1,412 / 1,575 ms	0.64 / 0.72	0	89.3 s

The same pattern as the published tables, measured on the client. TTFA grows from the first doubling of load: it is 0.54 s at 16 requests and 1.1 s at 48. The RTF stays below 1 at every level, 0.64 at 64 requests (its p95 is 0.72), and no request underran, because the server sends 2 s pieces: a client that has received a 2 s piece holds 2 s of audio, and every next piece arrived long before it was needed. The total audio the card produced grew from 6.8 s per second at one request to 89 s per second at 64, so throughput kept rising while each listener waited longer for the first sound. A 1 s TTFA target puts the knee between 32 and 48 requests. The knee of the RTF, and the first underruns, lie beyond the 64 requests this server admits: the next section removes the limit to find them.

The same sentence did not always come back at the same length. Across all levels, the longest output of a sentence was a median 1.47 times its shortest, and one 11-word English sentence came back at lengths from 5.4 to 14.5 s. Whisper recognized every word of even the longest of them (it wrote "Straightway" as "Straight away"), so the intelligibility scores below cannot see this, and "audio produced per second" counts the extra audio as if it were useful.

Measured: where the server itself overloads

At 64 requests the RTF was still 0.64, so the first table never reached the point where the server cannot keep up. I raised the server's own limit from 64 to 192 concurrent sequences, in both pipeline stages (code/tts/gpu/make_deploy_config.sh, code/tts/gpu/run_concurrency_cap192.sh), and repeated the closed loop at higher levels. The runs saved no audio to disk, so the client's own loop did not delay itself (its worst 1% of delays was under 5 ms at every level). Level 64 is the control: four visits gave a median RTF of 0.619 to 0.630, against 0.64 in the first table (code/tts/results/ch14_qwen3tts_concurrency_cap192_clear.json, _clear2.json, _clear3.json and _clear4.json; code/tts/ch14_harness/check_cap192_gates.py reads the checks that each level passed).

Concurrent requests	Requests	TTFA p50 / p95	RTF p50 / p95	Requests that underran	Audio produced per second
64	192	1,406 / 1,576 ms	0.619 / 0.693	0	92.3 s
80	240	1,713 / 1,833 ms	0.771 / 0.853	0	94.5 s
96	288	1,940 / 2,164 ms	0.875 / 0.975	0	98.8 s
112	336	2,234 / 2,433 ms	1.006 / 1.116	1	99.7 s
128	384	2,548 / 2,775 ms	1.152 / 1.279	205 (53%)	100.9 s
160	480	3,163 / 3,618 ms	1.421 / 1.587	321 (67%)	103.8 s
192	576	61 requests failed	—	—	—

The median RTF crosses 1 between 96 requests (0.875) and 112 (1.006). The first underrun is at 112, one request of 336; at 128 and 160 more than half the requests ran dry. The audio produced per second stops growing at about 100 s: 98.8 at 96 requests, 99.7 at 112, 103.8 at 160. A listener plays one second of audio per second, so a server that makes 100 s of audio per second can feed about 100 of them, and that is where the median RTF reaches 1. Above it, pieces arrive more slowly than they play, and the player runs out. TTFA keeps growing with the load, from 1.4 s at 64 requests to 2.2 s at 112 and 3.2 s at 160.

So the knee of the RTF is now the place where the server itself runs out of speed, not the place where its limit stops admitting requests. At 192 the KV cache runs out: the talker's cache holds 15,136 tokens at this setting (18,400 at a limit of 64), and 61 of the 576 requests failed: 24 with HTTP 500, "Request was preempted", and 37 whose response was cut off before its last piece. The client's loop delay at this level was 2.4 ms, so the failures are the server's. No RTF is reported for that level. Chapter 16 sends the same server an open-loop load, where requests arrive whether or not earlier ones have finished.

Intelligibility: Seed-TTS-eval

The standard way to score what a TTS system said is to transcribe it with a recognizer and compute WER against the input text. Seed-TTS-eval is the common protocol: 1,000 English samples from Common Voice and 2,000 Mandarin samples from DiDiSpeech (DiDiSpeech-2 in the seed-tts-eval README), plus a hard Mandarin set of "400 sentences with especially challenging patterns for autoregressive models such as word repetitions, tongue twisters" (Seed-TTS, arXiv:2406.02430, Secs. 3.1 and 4.2). English is transcribed by Whisper-large-v3 and Mandarin by Paraformer-zh.

Read its code before you quote a number from it (BytedanceSpeech/seed-tts-eval at commit 752f429):

• The Mandarin metric is a character error rate. The script splits Chinese into characters (truth = " ".join([x for x in truth])). CosyVoice 2's Table 6 heads the hard column "WER", its Table 8 heads the same numbers "CER".
• It averages per-utterance rates. wer = round(np.mean(wers)*100,3) in average_wer.py. Chapter 9 explained why that overweights short utterances: one error in a 2-word utterance and none in a 20-word one give 25% this way and 4.5% pooled.
• Missing outputs are skipped. if not os.path.exists(os.path.join(wav_dir, utt + '.wav')): continue. A system that fails on the hardest sentences and writes no file is not penalized. Count the files.
• No number or abbreviation normalization. Punctuation is stripped and English lowercased, nothing else; a system that reads "3" correctly is scored against the text as written.
• Neither recognizer is pinned to a revision, the scripts need an ARNOLD_WORKER_GPU variable and call sudo, and cal_wer.sh calls eval/average_wer.py although the file is at the repository root. The repository has no licence file.

The book's evaluator (code/tts/ch14_harness/eval_intelligibility.py) follows the protocol's recognizers but pools the edits, records the revisions of the recognizers it loaded, and scores the original human recordings of the same sentences. The human rows show how many errors the recognizers themselves make on this text: a difference between two systems smaller than that is not evidence. They are not a lower bound; synthetic speech can be easier to recognize than the recording, as Kokoro's English row below shows.

The CPU track, scored this way (code/tts/results/ch14_intelligibility_cpu.json): the 20 English and 20 Mandarin sentences of the book's text set, and the 5 three-sentence paragraphs per language synthesized the three ways of chapter 12. The human rows are the original recordings of the same sentences (LibriTTS-R and AISHELL-1).

Audio	Sentences	WER (English)	CER (Mandarin, Kokoro v1.1-zh)
human recordings	20 + 20	1.09% (4 of 366 words)	0.33% (1 of 304 characters)
Kokoro, one sentence per request	20 + 20	0.82%	0.99%
Kokoro, paragraph in one call	5 + 5 paragraphs	0.38%	1.72%
Kokoro, paragraph sentence by sentence	5 + 5 paragraphs	0.38%	0.43%
Kokoro, paragraph, text streamed in	5 + 5 paragraphs	0.38%	0.43%

On this text set Kokoro is about as intelligible to the recognizers as the human recordings: a handful of errors in a few hundred words or characters either way, too few to rank. Synthesizing sentence by sentence did not cost intelligibility; in Mandarin the whole-paragraph call made more errors (4 against 1 of 232 characters). Kokoro's older v1.0 Mandarin voice scored 2.96% on the single sentences (5 of 20 utterances with errors) against 0.99% for v1.1-zh.

The GPU track's audio, scored the same way (code/tts/results/ch14_intelligibility.json; the recognizers ran on the GPU, and their human and Kokoro rows match the CPU run above):

Qwen3-TTS audio	WER (English)	CER (Mandarin)	Utterances en / zh
sentences, 1 request at a time	1.91%	1.97%	20 / 20
sentences, 16 concurrent	1.40%	0.82%	24 / 24
sentences, 64 concurrent	1.66%	1.03%	96 / 96
paragraphs, all text at once	0.38%	0.43%	10 / 10
paragraphs, streamed in, sentence split	1.13%	1.72%	10 / 10

Load did not hurt intelligibility: from 1 to 64 concurrent sentence requests the rates stayed between 0.4% and 2.3%, with 2 to 21 utterances with any error per group. These rows show that the audio stays intelligible; they hold too few errors to rank the conditions, or to rank them against the human recordings' 1.09% and 0.33%.

Speaker similarity and naturalness

Speaker similarity (SIM) is the cosine similarity between speaker embeddings of the synthesized clip and the voice prompt. Seed-TTS-eval uses a WavLM-large speaker-verification model. The number depends on the embedding model: the same CosyVoice 2 Mandarin output scores 0.748 with WavLM and 0.806 with ERes2Net, and the authors note that "the results of speaker similarity are not consistent on different SV models" (arXiv:2412.10117, Table 6, p. 11, and p. 12). Compare SIM only within one embedding model.

Naturalness is ultimately a listening test: a mean opinion score (MOS) from human raters. Neural MOS predictors are cheaper, and each was built for a domain:

• UTMOS was built for the VoiceMOS Challenge 2022, on ratings of synthetic speech from the Blizzard and Voice Conversion challenges (arXiv:2204.02152).
• DNSMOS P.835 was built to evaluate noise suppressors and predicts speech quality, background noise and overall quality (arXiv:2110.01763).
• NISQA targets "distortions that occur in communication networks" and predicts overall quality plus noisiness, coloration, discontinuity and loudness (arXiv:2104.09494).

Use a predictor on the kind of audio it was trained on, as a screen, and confirm differences that matter with listeners. None of these papers shows that a predictor detects the chunk-join artefacts of chapter 13; the listening clips of that chapter are a better test for those.

The streaming-versus-offline delta

The quality question specific to streaming is: how much worse is the same model when it streams? CosyVoice 2 answers it for its own model on Seed-TTS-eval (arXiv:2412.10117, Tables 6 and 8):

Test set	Offline	Streaming	Difference
test-zh (CER)	1.45	1.45	0
test-en (WER)	2.57	2.38	−0.19
test-hard (CER)	6.83	8.08	+1.25

On ordinary sentences streaming costs nothing measurable; on the hard set it costs 1.25 points, mostly from the streaming language model (chapter 12). The table's streaming setting uses 15-token chunks, not the released code's 25. Measure the delta for your own system on the same text with the same scorer, and report both numbers.

Common misconceptions

"First-packet latency is TTFA." Most published first-packet numbers stop inside the server, before transport, before the client, and some are formulas. Measure on the client.

"Seed-TTS-eval WER for Mandarin." It is a CER, averaged per utterance, with missing files skipped.

"TTFA at one request is the server's TTFA." On one L20, CosyVoice 2's first chunk went from 220 ms at one request to 1,107 ms at four.

Try it

CPU track

1. python ch12_kokoro_cpu/kokoro_stream.py --threads 4 from code/tts/ records every chunk's arrival time and duration in its raw rows. Find the underrunning paragraph and recompute its deadline by hand.
2. Feed the three arrival times in figures/ch14/data/underrun.json to hbtts.stream.simulate_playback, with n_samples = each audio_s × 24,000, sr=24000 and prebuffer_s=1.5. Does the underrun disappear, and what did it cost in time to the first sound? Find the smallest pre-buffer that removes it.

GPU track

code/tts/gpu/run_concurrency_gpu.sh runs the Qwen3-TTS concurrency measurement, and code/tts/gpu/run_concurrency_cap192.sh repeats it with the server's limit raised to 192 (about 12 minutes; it saves no audio, so the client's loop does not delay itself; read the checks with check_cap192_gates.py). The ch14_harness/run_eval_gpu.sh script named in code/tts/README.md scores all runs, human recordings included. Compare the TTFA at one request with the paper's 101 ms, and list every difference in what the two numbers measure.

English and Mandarin

The evaluator scores English with Whisper's English normalizer and Mandarin as characters after the book's Mandarin normalizer, which keeps Chinese characters, Latin letters and digits and converts no numerals. Rescore one Mandarin run's transcripts with chapter 9's zh-full normalizer (hbasr.textnorm.ZH["full"], which turns "3" into "三") and see how many errors were only a numeral written two ways.

Key terms

• time to first audio (TTFA): here, from sending the request (or first text) to the arrival of the first audible sample, on the client.
• inter-arrival time: time between the arrivals of consecutive chunks.
• chunk deadline: TTFA plus the duration of all earlier chunks; a later arrival is an underrun.
• underrun / stall: the player has nothing to play; stall is the total silence it causes.
• pre-buffer: a wait after the first audio before playback starts, traded against TTFA.
• Seed-TTS-eval: the common intelligibility and similarity protocol for zero-shot TTS.
• speaker similarity (SIM): cosine similarity of speaker embeddings; depends on the embedding model.
• MOS / MOS predictor: human mean opinion score, and models that predict it.

Sources

• Du et al., "CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models", arXiv:2412.10117; FunAudioLLM/CosyVoice (commit 074ca6d): cosyvoice/cli/cosyvoice.py, runtime/triton_trtllm/client_grpc.py, runtime/triton_trtllm/README.Cosyvoice2.Unet.md, runtime/triton_trtllm/README.Cosyvoice3.md.
• Hu et al., "Qwen3-TTS Technical Report", arXiv:2601.15621.
• Zeghidour et al., "Streaming Sequence-to-Sequence Learning with Delayed Streams Modeling", arXiv:2509.08753.
• Torgashov et al., "VoXtream: Full-Stream Text-to-Speech with Extremely Low Latency", arXiv:2509.15969.
• Défossez et al., "Moshi: a speech-text foundation model for real-time dialogue", arXiv:2410.00037.
• Anastassiou et al., "Seed-TTS: A Family of High-Quality Versatile Speech Generation Models", arXiv:2406.02430; BytedanceSpeech/seed-tts-eval (commit 752f429).
• Saeki et al., "UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022", arXiv:2204.02152.
• Reddy et al., "DNSMOS P.835: A Non-Intrusive Perceptual Objective Speech Quality Metric to Evaluate Noise Suppressors", arXiv:2110.01763.
• Mittag et al., "NISQA: A Deep CNN-Self-Attention Model for Multidimensional Speech Quality Prediction with Crowdsourced Datasets", arXiv:2104.09494.