Turn taking & barge-in

Two decisions separate a voice agent that feels like a conversation from one that feels like a kiosk: when the caller has finished speaking, and what counts as an interruption. Both are configuration on the assistant. Both have defaults chosen against real calls. Both are worth changing for your use case.

Tune this before you tune anything else. It is the difference callers actually hear.

stt partials→endpointing decides→the model starts

Two vocabularies, and you need the right one

This is the single most confusing thing about configuring an assistant, so it goes first.

Write flat snake_case keys through the REST API and the dashboard.start_speaking_plan.on_punctuation_seconds — flat, one level, no nesting.

Read nested camelCase in @callagent/contracts.startSpeakingPlan.transcriptionEndpointing.onPunctuationSeconds — that is what the gateway receives after the control plane converts.

The converter reads the flat keys and nothing else. A nested block sent to the API is stored, returned by GET, and never reaches the gateway. It will look like it worked.

Every example on this page shows the flat shape, because that is the one you write.

When has the caller finished?

The four waits

The gateway picks one wait per turn, based on what the caller's last words looked like.

Wait	When it applies	Default	Range
start_speaking_plan.wait_seconds	No final transcript yet — voice activity alone.	0.4	0–5
start_speaking_plan.on_punctuation_seconds	The transcript ends in ., ? or !.	0.1	0–3
start_speaking_plan.on_number_seconds	The transcript ends mid-digits.	0.5	0–3
start_speaking_plan.on_no_punctuation_seconds	It ends on neither.	1.5	0–3

The fourth is the point of the block. An unpunctuated ending means the transcriber itself does not think the sentence is finished, so wait longer than you would after a full stop. Raise this one when your callers trail off; lower it when the assistant feels sluggish after short answers.

on_number_seconds exists because people read numbers with gaps in them. "My order is four four seven… nine two" contains pauses that would end any other sentence. Half a second of extra patience while digits are being spoken is the difference between capturing an order number and asking for it twice.

Smart endpointing

A model-based judgement of whether the sentence sounds finished, on top of the silence rules.

Key	Default	Range
start_speaking_plan.smart_endpointing_provider	callagent	off · callagent · livekit · model
start_speaking_plan.completion_threshold	0.7	0–1
start_speaking_plan.smart_endpointing_ms	2000	100–10000

Only callagent is implemented

Choosing livekit or model logs a warning in the gateway and silently runs the built-in scorer anyway. They are enum values, not implementations.

off is meaningful and is not the same as omitting the block: an absent block means "use the default", and the default is on.

Custom rules

Override the wait for specific utterances:

{
  "start_speaking_plan": {
    "wait_seconds": 0.4,
    "custom_rules": [
      {
        "target": "customer",
        "type": "regex",
        "pattern": "\\b(let me check|one second|hold on)\\b",
        "timeout_seconds": 6
      }
    ]
  }
}

target is customer (default), assistant or both. type is regex (default), contains or exact. timeout_seconds is in seconds, 0–15, and is converted to the contract's milliseconds on the way to the gateway.

pattern and timeout_seconds are both required — a rule missing either is dropped silently. A regex that will not compile is dropped too, with a logged reason.

Endpointing strategy

Which mechanism decides, as opposed to how long it waits, lives on the transcriber:

transcriber.endpointing.type	Fires when
vad_timeout	The silence timer expires.
provider	The transcriber says the utterance ended. Never on a timer.
punctuation	Terminal punctuation, at the punctuation wait; otherwise the full silence wait.
semantic	A completeness predicate returns true; otherwise the full silence wait.
hybrid	Default. Whichever enabled signal fires first.

Also on that block: min_speech_ms (default 120 — a cough is not a turn) and max_wait_ms (default 20000).

max_wait_ms is 20 seconds, not three. It is a ceiling, not a target; the other rules end almost every turn long before it. A short ceiling is what produces an assistant that interrupts a caller reading something off a screen. If you set it at or below your longest wait, the ceiling always fires first and your wait settings decide nothing — the gateway logs a warning when it sees that.

There is no form field for transcriber.endpointing.silence_ms

silence_ms, punctuation_seconds and number_grace_ms exist on the transcriber block in the contract and the control plane never writes them. The four waits above are where the durations come from. If you see those older names in a schema dump, they are the same knobs under a name nothing sets.

When does the assistant stop?

While the assistant speaks, the transcriber keeps running. stop_speaking_plan decides which of that incoming speech is an interruption.

Key	Default	Range	What it does
enabled	true		false is expressed as a barge-in nobody can trigger, not as an absent plan.
num_words	0	0–10	Transcribed words required before stopping. 0 means voice activity alone.
voice_seconds	0.2	0–0.5	Voice must persist this long to count.
backoff_seconds	1	0–10	After being interrupted, wait this long before resuming.

num_words: 0 is deliberate. Any value above zero forces the gateway to wait for transcription before it can stop talking, which costs 200–400 ms on every interruption, including the urgent ones.

The decision, in order

1. Not currently speaking → no interruption.
2. The utterance contains an interruption phrase → interrupt immediately, skipping every gate below.
3. The audio looks like echo of what was just played → not an interruption.
4. The whole utterance is an acknowledgement phrase → not an interruption.
5. Voice has not lasted voice_seconds → not yet.
6. num_words > 0 and fewer words than that → not yet.
7. Otherwise → interrupt.

The two phrase lists

This is the part that most affects how a call feels.

Acknowledgement phrases do not interrupt. A caller saying "mm-hmm" while the assistant explains something is agreeing, not interrupting, and an assistant that stops dead every time is exhausting. Twenty by default:

i understand · i see · i got it · i hear you · im listening · im with you · right · okay · ok · sure · alright · uh-huh · mm-hmm · yeah · yes · yep · got it · gotcha · makes sense · sounds good

The match is on the whole utterance, normalised. "okay but wait" is not an acknowledgement and will interrupt.

Interruption phrases always interrupt, immediately, without waiting for num_words or voice_seconds — and ahead of the echo guard, so they work even when the acoustics are bad. Thirteen by default:

stop · shut up · shut · wait · hold on · hold up · excuse me · actually · no no · listen · hang on · one moment · let me

These match on a token boundary inside the utterance, not on the whole thing.

Tune the lists, not the thresholds

If your assistant feels twitchy, add to acknowledgement_phrases before you raise num_words. Raising num_words delays every interruption, including "stop".

Echo rejection

Separate from the plan, and not configurable per assistant. The gateway keeps a decaying peak of what it just sent and treats inbound audio more than 6 dB below it as echo, held for 600 ms — because the carrier buffers what it was handed — decaying at 20 dB per second. These are gateway environment settings, not assistant fields.

An interruption phrase beats the echo guard, on purpose.

What an interruption produces

assistant.speech.interrupted carries spokenText, remainingText, spokenMs, detectionMs and stopAudioMs. In-flight model and synthesis calls emit llm.aborted and tts.aborted with abortReason: barge_in.

The transcript keeps only what was actually said out loud: the message carries interrupted: true, so the model's context reflects what the caller heard, not what the assistant intended to say. Getting this wrong is the classic voice-agent bug — the model believes it delivered a sentence nobody heard, and answers the next question as though it had.

Three metrics: interruption_ms, barge_in_detection_ms and barge_in_stop_audio_ms, the last with a target of 120 ms.

The greeting

first_message_interruptions_enabled defaults to false and should usually stay there. Callers routinely talk over the first second of a connected call — they were mid-sentence with ringback, or they say "hello?" before the greeting finishes. Treating that as an interruption abandons the greeting for no reason.

Leave it false if your greeting carries a disclosure or a recording notice. See AI disclosure & consent.

Silence and duration

Field	Default	Range
silence_timeout_seconds	30	1–600 over the API, 5–600 in the contract
max_duration_seconds	1800	10 to the platform ceiling

The gateway enforces these under different names

CallSession does not read the assistant's silence_timeout_seconds and max_duration_seconds directly. It reads limits.maxSilenceSeconds and limits.maxDurationSeconds from the per-call config the control plane sends. The values originate from the assistant, but if you are debugging a call that ran past its limit, look at the call config's limits block.

Silence is not always an error, so message_plan fills it before the timeout does: idle messages are spoken after an idle timeout (default 10 s), at most three times, with the counter reset when the caller speaks.

A complete, copyable block

{
  "transcriber": {
    "provider": "deepgram",
    "model": "nova-3",
    "endpointing": { "type": "hybrid", "min_speech_ms": 120, "max_wait_ms": 20000 }
  },
  "start_speaking_plan": {
    "wait_seconds": 0.4,
    "smart_endpointing_provider": "callagent",
    "completion_threshold": 0.7,
    "smart_endpointing_ms": 2000,
    "on_punctuation_seconds": 0.1,
    "on_no_punctuation_seconds": 1.5,
    "on_number_seconds": 0.5,
    "custom_rules": [
      { "target": "customer", "type": "contains", "pattern": "let me check", "timeout_seconds": 6 }
    ]
  },
  "stop_speaking_plan": {
    "enabled": true,
    "num_words": 0,
    "voice_seconds": 0.2,
    "backoff_seconds": 1,
    "acknowledgement_phrases": ["okay", "right", "mm-hmm", "sure", "got it"],
    "interruption_phrases": ["stop", "wait", "hold on", "actually"]
  },
  "first_message_interruptions_enabled": false,
  "silence_timeout_seconds": 30
}

PATCH that onto an assistant, publish, and place a call.

Next

• How a call works — where endpointing sits in the pipeline.
• Assistants — the rest of the configuration.