18. The voice cascade and its latency budget, then full duplex

What you will learn

• How a streaming recognizer, a language model and a streaming synthesizer are chained into a voice agent, where each of the roughly 1.3 seconds of a typical turn goes, and which part a speech engineer controls.
• How turn detection decides when the user has finished, and why its defaults matter more than the recognizer's look-ahead.
• How interruptions ("barge-in") are handled, and why the conversation history must record only what the user actually heard.
• What full-duplex speech-to-speech models change, and how to read their latency claims.

Before you start

• Chapters: 3 (the parts of a latency budget), 7 (endpointing and turn detection), 9 and 14 (measuring ASR and TTS), 15 and 16 (serving them), 17 (transport and the client). This chapter assembles them.
• Tools: the cascade in code/serve/ch18/ (see Try it) runs a streaming recognizer, a small local LLM and a streaming TTS model on one RTX 4090.

One turn, end to end

A voice cascade is three models in a row. A streaming recognizer turns the user's speech into text; a language model turns that text into a reply; a streaming synthesizer turns the reply into speech. Around them sit the transport and devices of chapter 17, and between them sit two decisions that are easy to overlook: when the user's turn is over, and when enough of the reply exists to start speaking it.

The number that users feel is voice-to-voice latency: the time from the moment the user stops speaking to the moment they hear the agent start. The practical way to measure it, from the guide by the developers of the Pipecat framework, is blunt and reliable: "record the conversation, load the recording into an audio editor, look at the audio waveform, and measure from the end of the user's speech to the beginning of the LLM's speech." Most tools report only the time to the first audio byte, which leaves out "phrase endpointing variation and network round-trip time" (Voice AI & Voice Agents guide, §5.1, fetched 2026-10-29).

Frameworks measure parts of it. LiveKit Agents, for example, computes an end-to-end figure as the time from the user's stopped_speaking_at to the agent's started_speaking_at (livekit-agents/livekit/agents/voice/agent_activity.py, lines 3417-3420, at livekit-agents@1.8.10). That is a server-side clock: the user's microphone, the network and the speaker are outside it.

The 1.3-second budget

The same guide publishes a full budget for one round trip, from a macOS laptop through WebRTC to cloud services and back (§5.1):

#	Stage	ms
1	macOS mic input	40
2	opus encoding	21
3	network stacks and transit	10
4	packet handling	2
5	jitter buffer	40
6	opus decoding	1
7	transcription and endpointing	300
8	llm ttfb	650
9	sentence aggregation	20
10	tts ttfb	120
11	opus encoding	21
12	packet handling	2
13	network stacks and transit	10
14	jitter buffer	40
15	opus decoding	1
16	macOS speaker output	15
	Total	1,293

The sixteen rows sum to 1,293 ms. (The prose just above the table on the same page says "the total is about 1,200 ms"; quote the table.) The guide calls 1,500 ms "an important target to aim for" and says it has shown Pipecat agents as low as 500 ms "by hosting all models within the same GPU-enabled cluster".

Group the rows by who owns them:

• Speech recognition and synthesis: transcription and endpointing 300, sentence aggregation 20, TTS first byte 120. Together 440 ms, 34%.
• The language model: first byte 650 ms, 50%.
• Transport and device (chapter 17): the other twelve rows, 203 ms, 16%.

Two of those rows deserve a closer look, because the table hides how much freedom there is in them.

"Transcription and endpointing: 300 ms"

This row is mostly not transcription. A streaming recognizer has already been transcribing while the user spoke (chapters 4-6); when the user stops, only the last chunk or two remain. What the row really holds is the endpoint delay: the time the system waits after the last word before it decides the user is done (chapter 7).

Now compare 300 ms with the defaults of real frameworks:

• LiveKit Agents waits at least min_delay 0.5 s and at most max_delay 3.0 s by default; with a streaming turn-detector model, 0.3 s and 2.5 s (livekit-agents/livekit/agents/voice/turn.py, lines 121-147 and 300-311). Its Silero voice activity detector defaults to min_silence_duration 0.55 s (livekit-plugins-silero/.../vad.py, lines 63-71).
• Pipecat's VAD defaults at v1.15.0 are confidence 0.7, start 0.2 s and stop 0.2 s (src/pipecat/audio/vad/vad_analyzer.py, lines 25-28). Its default end-of-turn decision, since v0.0.102, is not VAD silence at all but the Smart Turn v3 model, with a 3 s silence fallback (src/pipecat/turns/user_turn_strategies.py, lines 419-428; base_smart_turn.py, lines 26-29). (The guide itself still prints VAD_STOP_SECS = 0.8, the value the code had up to v0.0.101.)

With a 0.5 s minimum silence, the endpoint delay alone exceeds the whole 300 ms row. At LiveKit's streaming-detector minimum of 0.3 s, it fills the row and leaves nothing for the final recognition result. Only with a model-based turn detector, such as Pipecat's 0.2 s VAD stop plus a classification that the guide puts at "~30ms on a typical cloud vCPU" (§5.8.4), does the row have room to spare. So the 300 ms in the table assumes a learned turn detector. With a plain silence timeout, budget at least 500 ms for this row.

This is the claim chapter 3 made in general and the budget makes concrete: on a measured on-device recognizer, the endpointer lag was "almost ≥ 50% of the overall UPL" (Shangguan et al., arXiv:2104.02207, §3.2). The recognizer's look-ahead (0 to 1,040 ms for the settings of chapter 5) does not add to it either, as long as it is shorter than the endpoint delay: after the user's last word, the recognizer waits for its look-ahead of audio during the same silence the endpointer is already waiting through. At the GPU track's [70,13] and [56,13] settings, with 1,040 ms of worst-case look-ahead, that condition fails for every median endpoint delay in this chapter's table, and the look-ahead does add to the wait.

"Sentence aggregation: 20 ms"

Most TTS services synthesize better from a whole sentence than from a stream of words, so cascades collect the LLM's tokens until a sentence boundary. Pipecat does this by default (TextAggregationMode.SENTENCE, src/pipecat/services/tts_service.py, lines 326-327), and its own docstring prices it: "Produces more natural speech but adds latency (~200-300ms per sentence)" (lines 85-96).

Neither the guide nor Pipecat's docstring defines its number, so this is my reading of them, supported by the tutorial below, which measured 143 ms of "sentence detection": the 20 ms in the table is the time to detect the boundary once the sentence's tokens exist, and the 200-300 ms in the docstring is the time the LLM takes to produce the first sentence after its first token. The table's 650 ms LLM row is a first-byte time, not a first-sentence time, so a cascade that aggregates sentences pays part of that wait on top of it. The tutorial in arXiv:2603.05413 measured it as "Sentence detection 143ms" (Table 4, p. 9).

The alternative, streaming tokens into the synthesizer as they arrive, is what the dual-streaming models of chapter 12 are built for. CosyVoice 2 writes the cascade's first-audio bound as L_Chat ≤ N · d_llm + L_TTS, where N is the number of text tokens the synthesizer needs before its first audio and d_llm is the LLM's time per token; in its streaming mode N = 5 text tokens are interleaved with M = 15 speech tokens (arXiv:2412.10117, §2.5 p. 7 and §2.3 p. 4). If the LLM produces a token every 20 ms, waiting for five tokens costs 100 ms instead of a sentence.

Reading someone else's latency number

Published voice-agent numbers rarely start and stop at the same moments. The tutorial above reports "a measured time-to-first-audio of 755ms (best case 729ms)" (abstract). Its own Table 4 shows the 755 ms is exactly LLM first token 296 + sentence detection 143 + TTS synthesis 316 ms. Its code starts the clock with the transcript already in hand and stops it when the first sentence has been synthesized in full (chapters/04_streaming_pipeline/pipeline.py, lines 97-168 and 214-225, commit 597e01d). So the number contains no recognition, no endpoint delay, no user-side transport and no playout (the HTTP round trips to its LLM and TTS services are inside it). On this book's clock, which starts when the user stops speaking, you would add at least the recognizer's share; with its cloud recognizer's (Deepgram) lowest reported p50, 337 ms, from audio to final transcript, that is about 1,092 ms, still before transport and the device. Adding the guide's 203 ms of transport and device gives about 1,295 ms, close to the guide's total by coincidence of two very different splits. That last figure is arithmetic across two sources, not a measurement.

The lesson is not that either source is wrong. It is that a latency number is only meaningful with its start event, its stop event and its statistic. Chapter 3's vocabulary exists for this.

Turn detection is a trade, not a setting

A shorter endpoint delay answers sooner and interrupts more often. Users pause mid-sentence to think, to breathe, or in the middle of a phone number; a system that answers at every pause takes the turn from them. A longer delay feels sluggish. The guide's 2026 stack uses three layers to get both: "Voice activity detection, with a short (200ms) trigger", a turn model that listens to the audio (Smart Turn, which "supports 23 languages" and is BSD-licensed), and an LLM tag that can tell the system to wait 5 or 10 seconds when the user clearly has more to say; "we're making turn decisions in ~250ms from the time a user pauses" (§5.8.4-5.8.5). These timings are the vendor's own.

Voice activity detection itself is cheap. Silero VAD, which most frameworks use, runs on 32 ms windows (512 samples at 16 kHz) and, per the guide, "normally takes less than 1/8th of a typical virtual machine CPU core" (§5.8.1). Pipecat's 0.2 s start and stop settings are six of those windows each: 192 ms of speech before it decides the user is speaking, and 192 ms of silence before it decides they stopped.

Barge-in

Barge-in is the user speaking while the agent is talking. A good agent stops at once. That requires two things the guide states directly: "you need every part of your pipeline to be cancellable. You also need to be able to stop audio playout on the client very quickly" (§5.9).

Pipecat shows how this looks in code. A detected interruption becomes an InterruptionFrame, a system frame that jumps every queue (src/pipecat/frames/frames.py, lines 1425-1431), while ordinary data and control frames "are cancelled by user interruptions" (lines 135-153). Every processor reacts by cancelling and restarting its processing task (frame_processor.py, lines 868-869 and 1161-1182), and the output transport throws away the audio it had queued (base_output.py, lines 609-621). Its output chunks are 40 ms by default (base_transport.py, line 117), so at most about one chunk is already on its way when the reset happens, plus whatever the client and network still hold.

Two mistakes are common here.

• Audio already sent faster than real time. Many TTS services deliver audio faster than it plays. If the client has buffered five seconds of speech, stopping the server stops nothing the user hears. "You'll have to manually stop playout and flush audio buffers" (§5.9).
• A conversation history that remembers what was generated, not what was heard. If the LLM wrote a three-sentence answer and the user interrupted after the first sentence, the next LLM call must not believe the user heard all three. The fix is to keep the TTS word timestamps and cut the assistant's message at the last word actually played: "Use these word-level timestamps to buffer and assemble assistant message text that matches the audio heard by the user" (§5.9.2). LiveKit does the same by storing the "synchronized transcript" of an interrupted speech instead of its full text (agent_activity.py, lines 3375-3400).

False interruptions come from coughs, keyboard clicks, background speech and, most embarrassingly, echo: "you've probably heard your bot interrupt itself right when it starts talking" (§5.9.1). The remedies are the echo cancellation of chapter 17 and a VAD that requires a minimum length of speech (LiveKit requires 0.5 s by default, turn.py, lines 190-198).

Full duplex: the cascade with the boundaries removed

A cascade is half duplex: someone speaks, then the other side. Each boundary between the three models, and the turn decision between the two speakers, costs time. A full-duplex model listens and speaks at the same time, as people do, and removes those boundaries by modelling both speakers' audio streams together. This section introduces the idea and how to read its numbers; it is a direction the field is moving in, not yet a replacement for a well-tuned cascade.

Moshi (Kyutai, arXiv:2410.00037) is the clearest example. It models two audio streams in parallel, the user's and its own, as tokens of the Mimi codec at 12.5 Hz: one frame every 80 ms, 8 codebooks of 2,048 entries, 1.1 kbit/s (§3.3, p. 11). Before its own audio tokens it predicts time-aligned text tokens of what it is about to say, which it calls an inner monologue (§3.4.4, p. 15). Its theoretical latency is "80ms for the frame size of Mimi + 80ms of acoustic delay" = 160 ms, "with a practical overall latency as low as 200ms on an L4 GPU" (README at v0.2.5). For comparison, the paper cites "the 230 ms average in natural conversations measured over 10 languages" (§1, p. 2).

That 160 ms is not comparable to the cascade's 1,293 ms. It is the time for the model to start producing audio once it has decided to speak; deciding when to speak is now inside the model. The Full-Duplex-Bench benchmark (arXiv:2503.04721) measures exactly that decision. In its Table III (p. 6), Moshi takes the turn after 0.265 s on average and responds to an interruption in 0.257 s, but it also takes the turn at 98.5% of the user's mid-sentence pauses in the synthetic pause test. Gemini Live takes the turn at only 25.5% of pauses, and needs 1.30 s to take a real turn. Fast takeover and premature takeover are the same behaviour, the full-duplex form of an endpoint delay set too short.

Other systems keep part of the cascade. GLM-4-Voice (arXiv:2412.02612) uses one LLM that alternates 13 text tokens with 26 speech tokens from a single-codebook tokenizer at 12.5 Hz and 175 bit/s; its first audio needs 13 text and 10 speech tokens, and the paper gives a latency formula but no measured milliseconds in that section (pp. 5-6). Qwen2.5-Omni (arXiv:2503.20215) splits a "Thinker" that produces text from a "Talker" that produces speech tokens from the Thinker's hidden states, decoded to audio by a sliding-window model with "a lookback of 2 blocks and a lookahead of 1 block" (§2.4, p. 5).

The serving stack matters as much as the model. The same tutorial measured Qwen3-Omni three ways (Table 3, p. 5): through Hugging Face Transformers, the full model took 145,694 ms to first audio; through a cloud realtime API, 702 ms; and through vLLM it could serve only the text half. And on cost, the guide estimates that an agent built on OpenAI's Realtime API "is 3 to 5 times more expensive than an agent built with GPT-4.1" (§5.2.3, a vendor estimate). The guide's advice (§5.2.3) is that the commercial speech-to-speech APIs it tested are slower in practice than a carefully tuned cascade and harder to control; Full-Duplex-Bench measures full-duplex systems only (one of them, Freeze-Omni, is a cascade with a turn-state classifier) and includes no half-duplex cascade like this chapter's, and Moshi's 0.265 s takeover shows what a native duplex model can do. Full duplex is worth watching.

Measured: a cascade on one RTX 4090

The book's cascade runs every model of this chapter on one RTX 4090 (16,934 MiB of its 23,028 MiB usable with all three servers loaded): Silero VAD on the CPU, then a silence rule or Smart Turn v3.2 for the endpoint, Nemotron 3.5 ASR streaming in 320 ms chunks, Qwen3-1.7B on vLLM (greedy, asked to reply in one or two sentences), sentence aggregation, and Qwen3-TTS on vLLM-Omni, sentence by sentence, into a virtual player (code/serve/ch18/cascade.py; results in code/serve/results/ch18_cascade_*.json). The user turns are 25 LibriSpeech and 25 AISHELL-1 test utterances, each followed by 3.5 s of near-silence; the LLM answers them as if a user had said them. The clock starts at the end of the user's speech (the last VAD window scored as speech) and stops when the first audible sample of the reply plays. The player is a virtual one on the same machine, so the transport and device rows of the 1,293 ms table are not in these numbers: add that table's 203 ms of transport and device for a real connection. So that the three servers fit on one card, vLLM-Omni's default deploy file was reduced to 4 sequences and a memory budget of 0.235 and 0.16 of the card for its two stages, about 40% together (code/serve/ch18/qwen3_tts_cascade.yaml), and vLLM serves Qwen3-1.7B with 0.267 of the card's memory (code/serve/ch18/run_cascade.sh); one conversation at a time needs no more.

Stage, p50 / p95	Silence 300 ms	Silence 500 ms	Silence 800 ms	Smart Turn v3.2
endpoint delay (end of speech to endpoint)	322 / 357 ms	514 / 546 ms	802 / 836 ms	294 / 3,010 ms
ASR final after the endpoint	25 / 43 ms	24 / 36 ms	25 / 37 ms	25 / 43 ms
LLM first token after the request	12 / 14 ms	12 / 16 ms	12 / 13 ms	12 / 20 ms
first sentence complete after the first token	75 / 118 ms	70 / 134 ms	73 / 131 ms	62 / 125 ms
TTS first audio after the request	307 / 396 ms	307 / 371 ms	308 / 364 ms	294 / 373 ms
first audible sample played after it arrives	10 / 20 ms	10 / 20 ms	10 / 20 ms	10 / 20 ms
voice-to-voice	759 / 856 ms	955 / 1,016 ms	1,239 / 1,324 ms	717 / 3,474 ms
users cut off early (of 50)	14	7	4	15

Turns that were cut off early are left out of the latency rows, so each column covers a different set of turns (36, 43, 46 and 35). Over the 33 turns that no setting cut off, the voice-to-voice medians are 760, 963, 1,237 and 716 ms, within 8 ms of the table. A repeat of the 500 ms run gave 942 / 1,014 ms and again 7 early cut-offs, and no turn of the repeat underran after its first audible sample.

Four readings.

1. The endpointer is the largest row. With a 500 ms silence rule it is 514 ms, 54% of the median voice-to-voice time. Every 200 to 300 ms of silence threshold moves the total by about the same amount, and trades it against users cut off: 14, 7 and 4 of 50 at 300, 500 and 800 ms. In English the shortest rule cut off 10 of 25 turns, in Mandarin 4 of 25.
2. TTS first audio is the second. 307 ms, 32% of the median, almost all of it the server's own pipeline (chapter 12 measured about 305 ms to the first audible sample at one request).
3. A small local LLM is almost free. Qwen3-1.7B's first token came 12 ms after the request on an otherwise idle card, and its first sentence 70 ms later. The 650 ms LLM row of the budget table is not a law of cascades; it depends on the model and where it runs.
4. Smart Turn did not help on this material. It ended 46 of 50 turns at its 200 ms trigger, which gave the lowest median, 717 ms, but it cut off 15 users, more than a 300 ms silence rule, and the 4 turns where it fell back to 3 s of silence set the p95 at 3.5 s. Its inference took 57 ms at the median on the CPU. These are read sentences from audiobooks and news, not conversation, and they do not end the way a turn in a conversation does; the numbers measure Smart Turn on read speech.

Barge-in, measured

In a separate run of 20 turns the cascade was given prompts that produce long answers, and the next user utterance started 0.5 s after the agent's first audio was heard (code/serve/results/ch18_cascade_bargein.json; echo was not simulated). The VAD detected the user's speech 97.4 ms after it began at the median (three 32 ms windows). The virtual player stopped at once, and the TTS stream was closed 1.1 ms after the detection (p95 2.4 ms). By then the LLM had already finished its answer in 18 of the 20 turns; in the other 2, its stream was closed 3.6 ms after detection and vLLM reported no running request 42 ms after it. So stopping is fast; what the user waits for is detection, about 100 ms, plus a real device's output buffer, which the virtual player does not have. In this run the TTS first audio took 612 ms at the median instead of 307 ms, with the LLM decoding a long answer on the same card at the same time; the run did not isolate the cause.

Common misconceptions

"The LLM is the whole problem; make it faster and the agent is fast." The LLM is half of the published budget. The other half is speech, transport and the device, and the single largest speech-side item is the endpoint delay, a setting rather than a model. A faster LLM behind a 0.8 s silence timeout still feels slow.

"A smaller look-ahead in the recognizer makes the agent answer sooner." After the user's last word, the recognizer's look-ahead is waited for during the silence the endpointer waits through anyway. While the look-ahead is shorter than the endpoint delay, shrinking it from 480 to 80 ms costs accuracy (chapter 5) and buys none of the time the user waits; it only makes partial results appear sooner while the user is still speaking.

"Time to first audio is the voice-to-voice latency." A TTS or pipeline "time to first audio" usually starts when its input text arrives and stops when audio is ready on the server. The user waits from the end of their own speech until sound reaches their ear. Check the start and stop events of every number you compare.

Try it

The cascade for this chapter is in code/serve/ch18/ (code/serve/README.md gives the exact commands of the runs above). It needs the GPU: the three servers share one RTX 4090 (23,028 MiB usable).

GPU track

1. Start the three servers and run one cascade with code/serve/ch18/run_cascade.sh <logdir> "python ch18/cascade.py --name my_sil500 --endpoint silence --silence-ms 500" from code/serve. Rebuild the budget table above from its output, row by row, with p50 and p95.
2. Run it with --silence-ms 300 and 800 and with --endpoint smart. Watch which rows move and how many users are cut off early.
3. Run --barge-in, which injects the next test utterance 0.5 s after the agent's first audio, and measure how long the agent keeps talking after that speech starts. Add your own device's output buffer to it.

CPU track

code/serve/ch18/mock_servers.py stands in for the LLM and TTS servers, and the sherpa-onnx server of chapter 15 for the recognizer, so the cascade's logic (endpointing, aggregation, barge-in) can be run and debugged on a CPU. Its timings are those of the mocks, not of models.

English and Mandarin

The cascade's results keep both languages apart (budget_by_lang in each result file). At the 300 ms silence rule, English turns were cut off 10 times in 25 and Mandarin turns 4 times; at 800 ms, 2 each. Look at which English sentences were cut, and where the pause that cut them was.

Key terms

• voice cascade: speech recognizer, language model and speech synthesizer in a row.
• voice-to-voice latency: time from the end of the user's speech to the start of the agent's speech, as heard.
• endpoint delay: the wait after the last word before the system decides the user's turn is over.
• turn detector: a model that predicts from audio (and sometimes text) whether the user has finished.
• sentence aggregation: collecting LLM tokens until a sentence boundary before synthesis.
• barge-in: the user speaking while the agent speaks; the agent should stop.
• full duplex: listening and speaking at the same time, modelling both speakers' streams together.
• takeover rate (TOR): in Full-Duplex-Bench, the share of test points at which a model takes the turn.

Sources

• Daily / Pipecat, Voice AI & Voice Agents guide (updated June 2026, fetched 2026-10-29): https://voiceaiandvoiceagents.com/
• Pipecat v1.15.0 (commit fda1d5e0): https://github.com/pipecat-ai/pipecat/tree/v1.15.0
• LiveKit Agents 1.8.10 (commit ff962a01): https://github.com/livekit/agents/tree/livekit-agents@1.8.10
• Building Enterprise Realtime Voice Agents from Scratch: A Technical Tutorial, arXiv:2603.05413, and its code (commit 597e01d): https://arxiv.org/abs/2603.05413, https://github.com/SalesforceAIResearch/enterprise-realtime-voice-agent/tree/597e01d899fe0ba7fec7eb90a5643ff64695ced0
• Shangguan et al., Dissecting User-Perceived Latency of On-Device E2E Speech Recognition, arXiv:2104.02207: https://arxiv.org/abs/2104.02207
• CosyVoice 2, arXiv:2412.10117: https://arxiv.org/abs/2412.10117
• Moshi, arXiv:2410.00037, and the moshi README at v0.2.5: https://arxiv.org/abs/2410.00037, https://github.com/kyutai-labs/moshi/tree/v0.2.5
• GLM-4-Voice, arXiv:2412.02612: https://arxiv.org/abs/2412.02612
• Qwen2.5-Omni, arXiv:2503.20215: https://arxiv.org/abs/2503.20215
• Full-Duplex-Bench, arXiv:2503.04721 (v3): https://arxiv.org/abs/2503.04721