How a voice agent knows you have stopped talking

A voice agent has exactly one hard decision to make, and it is not what to say. It is when to say it. Come in too early and you cut the caller off in the middle of a phone number. Wait too long and they think the line has dropped and say "hello?". That decision is called endpointing, and in CallAgent it is not one threshold but five strategies that can be combined.

What follows is what actually runs in packages/audio/src/endpointing.ts and in the gateway, with the defaults from the code. Where a figure is computed rather than written down as a constant, I say so.

The five strategies

An assistant carries a type field with one of:

The default is hybrid, and the reason is that none of the other four is good enough alone. vad_timeout on its own cannot tell "yes." from "yes, and I also want to". punctuation on its own never fires if the transcription engine does not emit punctuation, and several do not.

The defaults

From the contract schema, which is what gets stored on an assistant:

• silenceMs — 400 ms. How much silence counts as "finished".
• minSpeechMs — 120 ms. Below this it was not speech, it was a noise.
• maxWaitMs — 20,000 ms. The absolute ceiling on one turn.
• punctuationSeconds — 0.1 s. If the text ends in a full stop, a hundred milliseconds is enough.
• onNoPunctuationSeconds — 1.5 s. With no terminal punctuation, wait far longer.
• onNumberSeconds — 0.5 s. Ending on a digit almost always means there is more coming.
• waitSeconds — 0.4 s, the baseline pause before speaking.

maxWaitMs used to be three seconds. The comment in the code says why it is not any more: every caller utterance longer than three seconds was endpointed mid-word and answered half-heard.

Why the number rule matters most

The rule for digits is a regular expression looking for a digit followed by optional spaces, dots, commas and dashes at the end of the text — with one explicit exception: a terminal full stop wins. "at 7.30." is finished. "my number is 0740 619" is not.

Without that rule, any agent that asks for a phone number, a national ID or an IBAN cuts the caller off at the third group of digits. With it, the pause between groups is covered by the extra 500 milliseconds.

The completion score is a heuristic, not a model

CallAgent has something called smart endpointing, with a default threshold of 0.7. It is tempting to assume there is a model behind it judging whether the sentence is complete. There is not, and that is a decision rather than a shortcut: a model round trip here would cost more latency than the endpointing it is tuning saves.

What it actually is: a function that starts at 0.5 and adds or subtracts.

• +0.35 if the text ends in terminal punctuation.
• −0.2 if it ends on a digit.
• For a single word: +0.4 if it is in a list of words that stand alone, −0.35 if it is in a list of words that cannot end a sentence, −0.15 otherwise.
• For several words: −0.45 if the last word is one that cannot end a sentence, +0.1 if there are at least four words.

The result is clamped between 0 and 1. The word lists are bilingual. The ones that cannot end a sentence include, in English, "and", "but", "or", "so", "because"; in Romanian, "și", "dar", "sau", "că", "să", "de", "la", "cu", "pe". The ones that stand alone include "yes", "no", "ok", "thanks", "bye" and their Romanian equivalents.

Above the threshold, the wait shortens to the smaller of the current wait and the punctuation wait. Below it, the wait lengthens — but not indefinitely: at most 2,000 ms.

The order in which it is decided

When several signals are live, precedence is fixed. Top down:

1. Custom rules you define on the assistant (regex, contains or exact).
2. Smart endpointing.
3. The shape of the final transcript: punctuation, trailing digit, or plain text.
4. silenceMs.

The first rule that matches wins. If none does, it falls through one level.

What does not happen on the audio path

Two things are worth stating because they are counterintuitive.

First: the function that runs on the media clock, fifty times a second, does not classify text and does not evaluate rules. Classification happens only when a final transcript arrives. The reason is written in the code: a regex sweep and a string split fifty times a second would put work on the audio path for no new information.

Second: there is not a single setTimeout in the whole endpointer. The gateway already runs a 20 ms media clock; a second clock would only be a second source of drift.

The voice detector, and where 240 milliseconds comes from

Underneath endpointing sits a voice-activity detector with its own thresholds: 8 kHz sampling, 20 ms frames, a 9 dB threshold above the noise floor, an absolute floor at −55 dB, a minimum of 2 speech frames, and a hangover of 12 frames.

Those 12 frames × 20 ms are the 240 milliseconds that appear in the documentation. It is a computed figure, not a constant written anywhere — change the frame size and it changes.

The hangover is why the detector does not declare silence at the first gap between two words. It is also why the barge-in gate measures sustained voice while the agent was audible rather than time since the first sound: endpointing can fire after 100 ms of punctuation while the detector is still inside its own 240 ms hangover.

What to tune, and in what order

If the agent cuts people off: raise onNoPunctuationSeconds first, not silenceMs. The case that hurts is the one without punctuation.

If the agent is slow to answer "yes" and "no": the completion score already handles that, but only if smart endpointing is on. Check the provider — if it is set to anything other than callagent, the gateway logs a warning and runs the built-in heuristic anyway.

If the agent cuts numbers off: raise onNumberSeconds. The default 0.5 seconds is calibrated for digits dictated slowly; somebody reciting an IBAN from memory pauses for longer.

And always check that maxWaitMs is longer than the longest wait you have configured. If it is not, the turn ceiling fires before that wait can ever elapse — the gateway logs a warning for this one too, at the start of the call.