How a call works
One turn of a voice conversation passes through seven stages. Each has its own hue, and that hue is the same everywhere it appears — in this diagram, in the latency bar on the assistant workbench, in the call timeline and in the metrics tab — so after a day of using CallAgent the colour is the label and you stop reading the word.
The caller only experiences one number: the gap between finishing their sentence and hearing the first syllable of the reply. Everything below exists to make that gap small and to tell you which stage spent it.
The stages
transport — audio arrives
A Twilio media stream, a FreeSWITCH channel from your own SIP trunk, or a WebRTC session from a browser. Six transports are defined (twilio_media_stream, telnyx_media, freeswitch_esl, websocket, webrtc, sip) and they all normalise to the same internal format before anything else runs: 16 kHz mono PCM. G.711 µ-law/A-law and Opus are transcoded at the edge in packages/audio, so an 8 kHz carrier trunk and a 48 kHz browser meet an identical pipeline.
Measured as transport_connect_ms and call_setup_ms.
Only two of the six carry a real call today
A Twilio media stream works. A browser WebRTC session works for audio and barge-in. The FreeSWITCH path — and therefore every SIP trunk — dies at the media handoff, and Telnyx has no media-stream builder despite declaring the capability. See What is not built.
stt — speech becomes text
The transcriber streams partial hypotheses as the caller talks. Fourteen providers are defined in the contracts (deepgram, assemblyai, speechmatics, gladia, soniox, azure, google, openai, openai_realtime, elevenlabs, cartesia, talkscriber, xai, custom). Thirteen have an adapter in the gateway; three carry a reviewed wire format. See Providers.
A transcriber the gateway has no adapter for invalidates the whole call config, not just that field — which is why validate refuses one at publish.
stt_first_partial_ms is the metric that matters here, and the platform's target for it is 300 ms.
endpoint — deciding they have finished
This is the stage nobody expects and everybody feels. Answering too early talks over the caller; answering too late makes the assistant feel slow and stupid. CallAgent treats it as configuration rather than a constant — see Turn taking & barge-in for the plans, the defaults and why maxWaitMs is 20 seconds rather than three.
Five strategies exist: provider, vad_timeout, punctuation, semantic and hybrid, which is the default. Every detection emits an endpoint.detected event carrying the strategy that fired and the silence that triggered it.
llm — the model thinks
The conversation so far, the system prompt, the resolved variables, the tool definitions and any retrieved knowledge go to the model. Eighteen model providers are defined, seventeen have an adapter, and thirteen carry a reviewed wire format. custom_llm takes an OpenAI-compatible endpoint of your own.
Only the last context_window_messages messages are sent — default 60.
Two numbers are recorded separately, and the difference matters: llm_ttft_ms (time to the first token, target 600 ms) is what the caller feels, because synthesis starts on the first sentence rather than the last. llm_total_ms is what you are billed for.
numFastTurns and fastModel do nothing
They are meant to route the opening turns of a call to a smaller model. The gateway never reads either. Every turn uses model.model. See What is not built.
tool — the model acts
If the model emits a tool call, the turn pauses here. The tool may be an HTTP request to your API, a knowledge query, a transfer, a DTMF burst, a handoff to another assistant, a call into an MCP server, or a function executed in the visitor's own browser. messages configured on the tool are spoken while it runs, so the caller hears "let me look that up" instead of silence.
Bounded by maxToolIterations (default 5) and by the tool's own timeoutSeconds (default 20). Measured as tool_latency_ms, mcp_latency_ms and knowledge_latency_ms.
tts — text becomes speech
Synthesis is chunked, which is what makes a reply start before the model has finished writing it. The chunker breaks the model's token stream at clause and sentence boundaries: the first chunk may break after 12 characters, later chunks after 60, and a hard ceiling of 240 characters forces a break at the last whitespace whether or not there is a boundary. A newline is always a boundary.
Providers that accept incremental text keep one socket open for the whole turn; the rest get a request per clause.
The chunker is not configurable
voice_formatting on an assistant becomes the contract's chunkPlan, and the gateway never reads it. The numbers above are the built-in defaults and there is no field that changes them.
tts_ttfa_ms — time to first audio — has a target of 400 ms. Nineteen voice providers are defined, sixteen have an adapter, and two — ElevenLabs and OpenAI — carry a reviewed wire format. None has been run against a live vendor in CI.
playback — the caller hears it
Audio is paced back down the transport at real time, which is precisely why barge-in is possible: there is always unspoken audio in the buffer that can be discarded the instant the caller starts talking.
The number that counts
| Metric | Stage | Target |
|---|---|---|
transport_connect_ms | transport | — |
stt_first_partial_ms | stt | 300 ms |
llm_ttft_ms | llm | 600 ms |
tts_ttfa_ms | tts | 400 ms |
endpoint_to_audio_ms | end to end | 1200 ms |
barge_in_stop_audio_ms | interruption | 120 ms |
endpoint_to_audio_ms is the only one a caller can perceive directly: from the moment endpointing fires to the moment the first audio sample of the reply leaves the gateway. The five targets are defined once, in LATENCY_TARGETS_MS in @callagent/contracts, and the dashboard colours a turn against them.
Twenty-five metric names are recorded in total — the full list is CallMetricName in the contracts package, and GET /calls/{call}/metrics returns them per turn. Beyond the six above they cover the rest of the pipeline (stt_final_ms, llm_total_ms, tts_total_ms, s2s_ttfa_ms, s2s_total_ms), the tools (tool_latency_ms, mcp_latency_ms, knowledge_latency_ms), the handover (queue_wait_ms, operator_answer_ms, handoff_latency_ms, transfer_latency_ms) and the network (audio_jitter_ms, audio_packet_loss_pct).
Read a real call
Every one of these stages appears, timed and colour-coded, in the call timeline: GET /calls/{call}/events for the raw stream, GET /calls/{call}/metrics for the numbers, or the Timeline tab of a call in the dashboard.
Speech to speech
A model can in principle be given the audio directly, skipping the stt and tts stages. Four realtime providers are defined (openai_realtime, google_realtime, elevenlabs_realtime, custom_realtime) and the assistant has a speech_to_speech_model field for it.
This does not run
The live call session never reads speech_to_speech_model. There is a complete, unit-tested speech-to-speech pipeline in the tree and nothing constructs one. An assistant configured for it runs the ordinary transcribe → model → synthesise pipeline and reports the ordinary metrics.
Only one of the four providers has an adapter at all. See What is not built.
What happens around the turn
Before the first turn, the gateway asks the control plane for a call config: the frozen assistant snapshot, the resolved credentials, the MCP server list, the limits (max duration, max tool calls, remaining concurrency, credit seconds) and the storage destination for recordings. Nothing in that request repeats during the call.
After the last turn, the call is finalised: the concurrency slot is released, the transcript is assembled, artifacts are stored and per-component costs are computed. See Calls & artifacts.
Next
- Turn taking & barge-in — the endpointing plans in detail.
- Call states & events — the 17 states and 70 event types.
- Providers & fallbacks — choosing what fills each stage.