7. Endpointing, VAD and turn detection

What you will learn

• The three different questions behind "is the user done?": is there speech, has the utterance ended, and is it now my turn to speak.
• How a voice activity detector (Silero VAD) and a recognizer's own endpointer (sherpa-onnx) make that decision, with their default numbers.
• Why the endpoint wait is usually the largest single part of user-perceived latency: with sherpa-onnx's rules it is two to four times the model's chunk and look-ahead together.
• What learned turn detectors add, what they cost, and which defaults the common voice-agent frameworks ship.

Before you start

• Chapters: 3 (endpoint delay, UPL), 4 (blank outputs of a transducer), 6 (partial and final results).
• Tools: the ssh-asr environment from code/README.md. Everything in Try it runs on a CPU.

Three questions

A system that listens has to decide when to stop listening. That decision answers one of three questions, and it pays to keep them apart.

1. Is there speech in this piece of audio? A voice activity detector (VAD) answers it, every few tens of milliseconds, from the sound alone.
2. Has the utterance ended? An endpointer answers it: enough silence has followed speech that the recognizer should finalize its result. This is endpointing.
3. Has the user finished their turn? A turn detector answers it: the user is done and expects a reply. People pause in the middle of sentences, so silence alone answers this badly.

A dictation app needs only the first two. A voice agent needs the third, because answering during a mid-sentence pause is an interruption, and waiting too long after a finished question feels slow.

Voice activity detection with Silero VAD

The VAD used throughout this book is Silero VAD (v6.2.3, MIT licence; the README badge's alt-text wrongly says "CC BY-NC 4.0"). It is a small network, about 2 MB, that reads fixed 32 ms windows: 512 samples at 16 kHz or 256 at 8 kHz, the only two rates it supports (num_samples = 512 if sr == 16000 else 256, utils_vad.py, lines 112-114). For each window it outputs a probability that the window contains speech.

A probability is not yet a decision. Silero's streaming VADIterator turns it into one with hysteresis: two thresholds instead of one, so that a probability hovering near a single threshold does not flip the decision every window. With its defaults (utils_vad.py, lines 592-597):

• speech starts when the probability reaches 0.5;
• speech ends when the probability has stayed below 0.35 (the threshold minus 0.15) for 100 ms (min_silence_duration_ms);
• the reported boundaries are padded by 30 ms (speech_pad_ms).

The figure shows the first lesson of this chapter on real audio. The speaker pauses for about 290 ms in the middle of the sentence. With the default 100 ms minimum silence, the VAD reports an end of speech there, decided at 7,488 ms, and a new start at 7,600 ms. As a segmenter for transcription that is fine. As a turn detector it would have cut the speaker off with about a quarter of the sentence still to come (7 of 24 words, 2.7 of 10.1 s). The real end, at 10,100 ms, was decided at 10,272 ms.

So the minimum silence is the VAD's main dial. Raise it, and fewer pauses are taken for ends, but every real end is decided later by the same amount. That trade is measured below.

The recognizer's own endpointer

A streaming recognizer can endpoint without a separate VAD, from its own outputs. A transducer emits a blank when it has nothing new to say (chapter 4), and a run of blanks after some text is a sign that the speaker has stopped. sherpa-onnx's endpointer works exactly this way. It counts trailing blanks and converts them to time: trailing_silence_frames = num_trailing_blanks * 4 with a 10 ms frame shift, so each blank of the Zipformer's 40 ms output frames counts as 40 ms (online-recognizer-transducer-impl.h, lines 380-388). Any real token resets the count. The endpointer measures the model's silence, not the microphone's energy: a cough the model ignores is silence to it, and background speech it transcribes is not.

Three rules decide, and any one of them firing is an endpoint (endpoint.h, lines 41-52):

Rule	Fires when	Default
1	trailing silence, even if nothing was decoded	2.4 s
2	trailing silence after at least one token was decoded	1.2 s
3	the utterance has lasted this long, whatever else	20 s

Rule 2 is the one that ends a normal sentence. One detail catches people out: endpointing is on by default in the C++ API (bool enable_endpoint = true) but off in the Python API (enable_endpoint_detection=False), with the same rule values (online_recognizer.py, lines 112-115). A Python program that never sees an endpoint has usually not turned it on.

What rule 2 costs, worked out

Rule 2's 1.2 s is 30 consecutive blank frames of 40 ms. The model shows a word a median 582 ms after the speaker finished it (its word latency, chapter 9). The count of blanks starts then, so the endpoint fires about 582 + 1,200 = 1,782 ms after the speaker stopped. Audio arrives in chunks, so add up to one model chunk of waiting, 320 ms for the CPU track's Zipformer.

Compare that with what the model itself needs. Chapter 3 found about 458 ms of chunk and look-ahead for this model. The default endpoint wait alone is 2.6 times that. Tuning rule 2 moves the user's wait more than any change to the model's chunk size can.

Why the endpointer dominates

This is not a peculiarity of one toolkit. Shangguan et al. measured user-perceived latency on Android phones, one CPU core, with an endpointer that waited for 0.9 s of trailing silence. In their words, "EP lag is almost ≥ 50% of the overall UPL in a RNN-T model"; for one model, the endpointer's lag was 1.00 s of a 1.60 s median UPL (arXiv:2104.02207, Tables 1-2). Then they replaced the fixed silence rule (their Table 3, for alignment-restricted RNN-T models, median UPL and endpointer lag):

Endpointer	Median UPL	Median endpointer lag
fixed 0.9 s trailing silence	1.37 s	0.81 s
plus a neural endpointer	0.94 s	0.19 s
plus endpointing inside the recognizer (E2E)	0.87 s	0.13 s

Changing the endpointer cut the median wait by half a second. In the same paper, cutting the model's compute by 55% made the median UPL only 2% worse.

The recognizer and the endpointer can also be trained together. Bijwadia et al. trained one model to transcribe and to predict the end of the query, and at the same WER of 5.8 the median endpoint latency dropped from 390 to 270 ms, the 90th percentile from 740 to 570 ms (arXiv:2211.00786, Table 1). The paper's abstract puts it as "reduces median endpoint latency by 120 ms (30.8% reduction), and 90th percentile latency by 170 ms (23.0% reduction), without regressing word error rate".

Turn detection

An endpointer answers question 2 by waiting. A turn detector tries to answer question 3 by understanding: does what the user said so far, and how they said it, sound finished? "I'd like to book a table for" is unfinished whatever the silence; "Thank you." is finished almost immediately.

Two families are in common use, and both run alongside a VAD rather than replacing it:

• Text-based: a small language model reads the recognizer's transcript. LiveKit's text turn detector is a fine-tuned Qwen2.5-0.5B-Instruct that reads the last 6 turns (up to 128 tokens), runs as an INT8 ONNX model on a CPU, and covers 14 languages; its licence, the "LiveKit Model License", allows use "only ... together with the LiveKit Agents framework" (livekit/turn-detector model card). LiveKit's documentation gives its run time as "~50-160 ms" per turn on a CPU (docs.livekit.io, fetched 2026-10-29).
• Audio-based: a model listens to the end of the audio itself, including intonation. Pipecat's Smart Turn v3 is a Whisper Tiny encoder with a linear head, 8M parameters, an 8 MB int8 ONNX file, BSD-2-Clause, 23 languages per the pipecat-ai/smart-turn README (pipecat-ai/smart-turn-v3 model card).

For Smart Turn you will find "12 ms", "60 ms" and "450.6 ms" quoted for its run time. They are all correct and describe different things. Daily's announcement of v3 gives preprocessing plus inference per machine: v3 takes 12.6 ms on an AWS c7a.2xlarge, 59.8 ms on a c8g.medium, 94.8 ms on a t3.medium and 3.6 ms on an L4 GPU; the older, wav2vec2-based v2 took 450.6 ms on the c7a.2xlarge.

A turn detector can even predict the end before it happens. Zink et al. trained a model to recognize the end of an utterance with part of the future audio masked; it "can reasonably operate up to 300 ms before an utterance ends" (arXiv:2409.19990, Sec. IV); at 500 ms of masked audio its mean error was about 100 ms, against about 300 ms for the baseline (LibriSpeech 100 h, Fig. 4).

The defaults you will ship with

The voice-agent frameworks combine a VAD, an optional turn model and a fallback timer. Their defaults, read from the code of the versions named:

Framework	Decision	Default
Pipecat v1.15.0	Silero VAD stop	0.2 s (VAD_STOP_SECS = 0.2; releases up to v0.0.101 used 0.8 s)
Pipecat v1.15.0	then a user-speech timeout, or Smart Turn	0.6 s (user_speech_timeout), or the turn model's decision
LiveKit Agents 1.8.10	endpointing delay, VAD only or text turn model	min 0.5 s, max 3.0 s
LiveKit Agents 1.8.10	endpointing delay with the audio turn model	min 0.3 s, max 2.5 s
sherpa-onnx	rule 2 trailing silence	1.2 s

Sources: Pipecat src/pipecat/audio/vad/vad_analyzer.py lines 25-28 at v1.15.0; LiveKit livekit-agents/livekit/agents/voice/turn.py lines 135-147 at livekit-agents@1.8.10; sherpa-onnx endpoint.h lines 51-52. In LiveKit, the minimum delay is how long it waits after the turn model says "finished", and the maximum is how long it waits when the model says "not finished" before ending the turn anyway.

Every one of these is larger than the 160 ms chunk of the running example. Chapter 18 builds a cascade budget on them.

Measured: the endpoint trade-off

The trade is the same for every endpointer: a shorter silence setting ends real turns sooner but takes more pauses for ends. I measured both sides for sherpa-onnx's rule 2 and for Silero VAD's minimum silence, on the same utterances, English and Mandarin (code/ch07_endpoint/endpoint_sweep.py). Each utterance gets 2 s of near-silence appended, so the endpoint can happen after the last word; the end of speech is where offline Silero VAD finds the last speech sample. Two numbers per setting:

• endpoint delay: from the end of speech to the first endpoint decision, over utterances whose first endpoint came after the end of speech;
• false cut-offs: utterances whose first endpoint came more than 50 ms before the end of speech, that is, a pause inside the sentence was taken for its end.

I used 100 utterances per language from the book's test subsets. They contain plenty of pauses: in English, 81 pauses of 200 ms or more inside utterances (21 of at least 500 ms, the longest 1.34 s); in Mandarin, 50 (16 of at least 500 ms, the longest 1.06 s). For English, the end of speech is the end of the last word in a forced alignment of the reference text, which is independent of both endpointers. For Mandarin the end of speech is where offline Silero VAD finds it; that reference is fair to sherpa-onnx but circular for Silero (its delay against its own offline decision is just its setting plus 32 to 60 ms), so Mandarin is shown for sherpa-onnx only. The delay is a median over the utterances that were not cut off, so it is computed from fewer utterances at the short settings (32 of 100 at 0.2 s for sherpa-onnx in English).

English, from the end of the last word (code/ch07_endpoint/results/endpoint_sweep.json):

sherpa-onnx rule 2	0.2 s	0.4 s	0.6 s	0.8 s	1.0 s	1.2 s
endpoint delay, p50	982 ms	1,262 ms	1,342 ms	1,622 ms	1,743 ms	1,963 ms
utterances cut off early (of 100)	68	30	18	5	4	3

Silero minimum silence	100 ms	200 ms	300 ms	500 ms	800 ms	1,000 ms
endpoint delay, p50	298 ms	400 ms	504 ms	696 ms	976 ms	1,200 ms
utterances cut off early (of 100)	52	43	29	15	5	2

Mandarin, sherpa-onnx rule 2, from the VAD's end of speech: p50 delay 600, 808, 953, 1,179, 1,309 and 1,592 ms at 0.2 to 1.2 s, with 95, 42, 17, 10, 5 and 1 utterances cut off early.

Three readings.

1. Every setting that cuts off few utterances waits about a second or more. To cut off at most 5 of 100 English utterances, Silero needed 800 ms of minimum silence (976 ms median delay) and sherpa-onnx 0.8 s (1,622 ms). This is the gap a turn detector is meant to close.
2. sherpa-onnx's delay is its setting plus about 0.8 s. The arithmetic of the worked example above explains it with measured parts: a word appears in the partial result a median 582 ms after it ends (the word latency of chapter 9, over all words), the blank count starts only then, and trailing blanks are counted only when a 320 ms model chunk has been processed. 0.2 s of setting plus 582 ms plus up to 320 ms of chunk spans 782 to 1,102 ms; the measured median is 982 ms. A VAD that listens to the audio directly starts counting at the end of the sound, which is why, at 5 cut-offs in 100, Silero ends the turn sooner. The comparison needs one correction: Silero decides on 32 ms windows, sherpa-onnx only when a 160 ms packet has been processed. Rounding each Silero decision up to the end of its 160 ms packet raises its delays by 56 to 80 ms (to 370, 480, 560, 760, 1,040 and 1,260 ms; code/ch07_endpoint/silero_packet_rounding.py); at 5 cut-offs in 100 Silero is then about 580 ms sooner (1,040 against 1,622 ms).
3. Mandarin cuts off more at short settings. At 0.2 s, 95 of 100 Mandarin utterances were cut off against 68 English ones, although the Mandarin set has fewer pauses of 200 ms or more. From 0.6 s up the two languages are close (17 against 18 at 0.6 s). Choose the setting per language.

Common misconceptions

"A VAD tells you when the user is done." It tells you when there is no speech. With its default 100 ms minimum silence, Silero VAD reported an end of speech in the middle of a sentence in the figure above.

"A faster model ends turns faster." The endpoint wait is set by a silence threshold or a turn model, and with sherpa-onnx's rules it is 2.1 to 4.3 times the model's whole chunk and look-ahead (458 ms), and with Silero's 500 ms setting 1.5 times.

"Endpointing is something the recognizer does for me." In sherpa-onnx's Python API it is off until you enable it, and its default rule 2 waits 1.2 s, far longer than a conversation tolerates.

Try it

CPU track

1. Run python figures/ch07/data/export_vad.py (with PYTHONPATH=code) and look at events in its output. Change min_silence_duration_ms in the VADIterator call to 300 and run it again: the mid-sentence end at 7.4 s disappears, and the real end is decided at 10,464 ms instead of 10,272 ms, 192 ms later. That is the trade in one utterance.
2. Run python code/ch07_endpoint/endpoint_sweep.py --n 100 to reproduce the trade-off table, then python code/ch07_endpoint/endpoint_vs_alignment.py for the English rows measured from the forced alignment. It streams each utterance through sherpa-onnx with simulated pacing, so it runs faster than real time.
3. In your own sherpa-onnx program, set enable_endpoint_detection=True and rule2_min_trailing_silence=0.6, and say a sentence with a pause in it. Count how often it cuts you off.

GPU track

None for this chapter: VADs and endpointers run on a CPU in every system in this book.

English and Mandarin

The sweep runs both languages on the same settings. Look for whether Mandarin sentences, with their different pause patterns, produce more or fewer false cut-offs at the same silence setting.

Key terms

• voice activity detector (VAD): decides, per short window, whether audio contains speech.
• hysteresis: separate start and end thresholds, so a decision does not flip on every window.
• minimum silence: how long speech must be absent before the VAD reports an end.
• endpointer, endpointing: the decision that an utterance has ended and its result should be finalized.
• trailing silence: in sherpa-onnx, the run of blank outputs since the last token, in seconds.
• turn detector: a model that decides whether the user has finished their turn, from text or audio.
• false cut-off: an endpoint decided at a pause inside the user's utterance.

Sources

• silero-vad v6.2.3 (commit 5cd7945): src/silero_vad/utils_vad.py, LICENSE, README.md: https://github.com/snakers4/silero-vad/tree/5cd7945676eb32225748052e2e6a0580e4686a08
• sherpa-onnx v2.0.1 (commit af2d8d8): sherpa-onnx/csrc/endpoint.{h,cc}, sherpa-onnx/csrc/online-recognizer-transducer-impl.h, sherpa-onnx/csrc/online-recognizer.h, sherpa-onnx/python/sherpa_onnx/online_recognizer.py: https://github.com/k2-fsa/sherpa-onnx/tree/af2d8d81dbfa61c4cea4c568da78f3982edca005
• Shangguan et al., "Dissecting User-Perceived Latency of On-Device E2E Speech Recognition", arXiv:2104.02207.
• Bijwadia et al., "Unified End-to-End Speech Recognition and Endpointing for Fast and Efficient Speech Systems", arXiv:2211.00786.
• Zink et al., "Predictive Speech Recognition and End-of-Utterance Detection Towards Spoken Dialog Systems", arXiv:2409.19990.
• livekit/agents, livekit-agents@1.8.10 (commit ff962a0): livekit-agents/livekit/agents/voice/turn.py: https://github.com/livekit/agents/tree/ff962a019b6500e08f8f1fcd7f55673361bd9e91
• livekit/turn-detector model card (revision fba34c3), Hugging Face.
• pipecat, tag v1.15.0: src/pipecat/audio/vad/vad_analyzer.py, src/pipecat/turns/user_stop/speech_timeout_user_turn_stop_strategy.py (line 51, user_speech_timeout): https://github.com/pipecat-ai/pipecat/tree/v1.15.0
• pipecat-ai/smart-turn (commit 4786657) and the pipecat-ai/smart-turn-v3 model card (revision f766f81).
• Daily, "Smart Turn v3" announcement, 2025-09-11.