The pipeline
What happens in the second after someone stops talking.
Seven stages, four latency targets, one event ledger. This page is the whole mechanism, including the parts that are still targets rather than measurements.
- transport audio in and out
- stt speech to text
- endpoint the caller stopped
- llm the model answers
- tool tools and MCP
- tts text to speech
- playback audio to the caller
Seven stages, and a colour each.
The colours below are not a chart theme. Each stage keeps the same hue everywhere it appears — the latency bar, the debug timeline, the metrics tab, the provider card you chose it on — until the colour stops needing to be read.
| transport | Audio arrives from a carrier as a media stream or over SIP and leaves the same way. The gateway holds the socket for the whole call and emits transport.connected and transport.disconnected around it. |
|---|---|
| stt | Streaming transcription. Partial hypotheses arrive continuously and are corrected in place; the final one is what the model is given. Transcribers come in three classes — streaming, buffered and batch — and only a streaming one may be published as the live transcriber. |
| endpoint | Deciding the caller has finished, which is a guess made under time pressure: too eager and you talk over someone drawing breath, too patient and every answer feels slow. Voice activity and punctuation both feed it. |
| llm | The model answers, streaming. Tools it decides to call, knowledge it retrieves and structured fields it fills all happen inside this stage, and the first token is what the clock is measured to. |
| tool | Server tools and MCP tools, each with a JSON schema, each called individually. A tool can be refused before it runs, which is recorded as tool.rejected rather than silently dropped. |
| tts | Speech synthesis, streamed sentence by sentence so playback can start before the answer is finished. The clock runs to the first frame of audio, not to the last. |
| playback | Audio going back to the caller, and the point at which it must stop the instant they interrupt. Everything the caller actually heard is measured here. |
Four numbers the system holds itself to.
A voice agent has one hard constraint: the pause between a person finishing and the agent starting. Past roughly a second it stops sounding like a conversation, so the pause is treated as a budget with named parts rather than as an emergent property.
Every turn is timed against these and the console flags the ones that miss. A missed budget is stated in words as well as in colour, because a red number on its own is invisible to a red-green colourblind supervisor watching a wallboard.
They are targets, not promises. On the first real call the 1,200 ms target was missed with 3,091 ms measured, and the console flagged the turn. That is what the contract looks like when it is not met.
- stt.first_partial
- 300 ms
- llm.first_token
- 600 ms
- tts.first_audio
- 400 ms
- endpoint_to_audio
- 1,200 ms
- barge_in.stop_audio
- 120 ms
from the caller starting to speak to the first partial transcript.
from prompt sent to the model's first token.
from text sent to the first frame of synthesised audio.
the whole contract: silence detected to audio leaving for the caller.
from the caller cutting in to the assistant's audio stopping.
Read from configuration when this page renders. Change the budget and this panel changes.[1]
Knowing when someone has actually finished.
Silence is not the signal people think it is. A caller pauses mid-sentence to think, breathes between clauses, and stops dead at the end of a short answer. An endpointer that fires on silence alone interrupts the first and dawdles on the third.
So two things run at once: voice activity detection with a hangover window, and a punctuation timer that fires quickly once the transcript looks finished. The overlap between them is deliberate and is the reason the barge-in gate measures sustained voice rather than time since the last onset.
| on_punctuation_seconds | 100 ms by default. Once the transcript ends in punctuation the endpointer fires this long after, without waiting for the voice detector to agree. |
|---|---|
| vad_hangover | 240 ms. The voice detector stays latched this long after the last speech frame, so a breath between clauses is not read as the end of a sentence. It routinely overlaps the endpointer firing, which is why elapsed time since the voice onset is a useless barge-in signal. |
| start_speaking_plan.wait_seconds | Default 0.4 s, range 0 to 5. How long the assistant waits after endpointing before it starts speaking, per assistant. |
The start-speaking plan set on an assistant is the one the call runs on: the gateway parses it at session start and its four waits — the plain one, and the ones for a transcript ending on punctuation, on digits, or on neither — decide when a turn is over.[2]
- 2.040 assistant.speech.started
- 2.600 user.speech.started the caller cuts in
- 2.720 assistant.speech.interrupted barge_in.stop_audio target
- 2.720 llm.aborted no further tokens are generated
The 120 ms is the configured target. The rest is the caller's own timing and asserts nothing.[3]
Stopping mid-word.
When the caller cuts in, three things have to happen together: playback stops, the model generation is aborted so no more tokens are paid for or spoken, and the assistant turn is recorded as interrupted so the transcript does not claim the caller heard the whole sentence.
The gate measures sustained caller voice heard while the assistant was audible, not time since the voice detector last fired. The difference is not academic: measured from the onset, the very first inbound frame of the assistant turn already clears a 200 ms gate, and the assistant gets cancelled after a word or two on every single turn.
| num_words | Default 0, range 0 to 10. Words of transcript required before an interruption counts. Zero means the voice detector decides; a non-zero value adds a few hundred milliseconds of transcription latency to every interruption, which the caller hears as being talked over. |
|---|---|
| voice_seconds | Default 0.2 s, maximum 0.5. Sustained caller voice, heard while the assistant was audible, before the turn is cancelled. |
| backoff_seconds | Default 1.0 s, range 0 to 10. How long the assistant stays quiet after being interrupted before it may speak again. |
70 things a call can say about itself.
This is the complete list, read from the enum when the page renders rather than typed out here. Every one carries a millisecond offset from the start of the call and an optional payload, and all of them reach you through the events endpoint and through webhooks.
They are grouped below by the stage that emits them, in the same colours. There is no separate summarised event stream for customers: the debug timeline an engineer opens at three in the morning is a straight render of these rows.
Grouped with the same rule the console's timeline component uses.[4]
transport 43
- call.created
- call.ringing
- call.connected
- call.ended
- call.failed
- transport.connected
- transport.disconnected
- call.state.changed
- s2s.started
- s2s.first_audio
- s2s.completed
- s2s.failed
- handoff.started
- handoff.completed
- transfer.started
- transfer.completed
- transfer.failed
- operator.queued
- operator.ringing
- operator.connected
- operator.rejected
- operator.missed
- operator.wrap_up
- queue.abandoned
- queue.overflowed
- supervisor.attached
- supervisor.mode_changed
- supervisor.detached
- dtmf.received
- dtmf.sent
- sip.request.sent
- sip.response.received
- voicemail.detected
- campaign.opt_out_detected
- recording.consent_announced
- recording.consent_recorded
- recording.started
- recording.paused
- recording.resumed
- recording.ready
- analysis.completed
- provider.fallback
- provider.error
stt 4
- user.speech.started
- user.speech.ended
- transcript.partial
- transcript.final
endpoint 1
- endpoint.detected
llm 5
- llm.started
- llm.first_token
- llm.completed
- llm.failed
- llm.aborted
tool 9
- tool.started
- tool.completed
- tool.failed
- tool.rejected
- mcp.started
- mcp.completed
- mcp.failed
- knowledge.queried
- knowledge.insufficient
tts 5
- tts.started
- tts.first_audio
- tts.completed
- tts.failed
- tts.aborted
playback 3
- assistant.speech.started
- assistant.speech.interrupted
- assistant.speech.completed
What a finished call leaves behind.
Each of these is a route on the call resource as well as a tab in the console, so anything you can read on a screen you can also fetch.
| GET /{call}/recording | Mono, stereo or dual-channel. Dual-channel keeps the caller on one channel and the assistant or operator on the other, time-aligned, so the two are separable afterwards — a stereo mix is spatial and is not. Delivered as a signed URL that expires. |
|---|---|
| GET /{call}/transcript | The live transcript, turn by turn, with interrupted turns marked as interrupted. This is the same text the model was given, not a cleaned-up version of it. |
| post-call transcription | A second, higher-accuracy pass run after hangup by a batch transcriber, which can be a different provider from the live one. The system refuses to publish a batch-only transcriber as the live one, because that mistake takes every call down at once. |
| GET /{call}/metrics | Per-turn latency broken down by stage, which is what the stacked bar in the console is drawn from. |
| GET /{call}/costs | Per-call cost accounting split by stage and provider, written to an immutable ledger rather than recomputed on read. |
| structured outputs | The structured fields you asked the model to fill during the call, validated against your schema and stored as versioned records. |
| GET /{call}/artifacts | Everything the call produced as files, on the platform bucket or on your own. |
| retention and erasure | A scheduled job applies your retention policy daily. Deleting a recording is a real deletion from storage; erasing the rest of a call runs through retention rather than through a button. |
Notes
-
1
Targets:
config/callagent.phpandpackages/contracts/src/metrics.ts, which hold the same five constants on either side of the wire. -
2
Endpointing and hangover figures:
apps/voice-gateway/src/session/barge-in.ts. The start-speaking plan is defined inapp/Filament/Resources/Assistants/Schemas/AssistantForm.phpand consumed byendpointingConfig()inCallSession.ts, which has its own test file. -
3
Stop-speaking plan defaults:
app/Filament/Resources/Assistants/Schemas/AssistantForm.php, the stop-speaking section. -
4
Event vocabulary:
app/Enums/CallEventType.php. The grouping rule is the one inresources/views/components/ca/timeline.blade.php.