Providers & fallbacks

Every pipeline stage is filled by a provider you choose, and the choice is a trade of latency against cost against quality. CallAgent puts all three numbers next to the provider at the moment you pick it, tells you which of them it can actually vouch for, and lets you order fallbacks for when one of them has a bad afternoon.

The registry

Providers are not a hard-coded list. They live in registry tables the platform keeps — llm_models, transcriber_models, voices, and their provider tables — each row carrying capability flags, price columns and published typical latency (typical_ttft_ms, typical_first_partial_ms, typical_ttfa_ms). Adding a model is a registry row, not a code change.

Read it through the API:

cURL

curl "$CALLAGENT_API_URL/registry/models" \
  -H "Authorization: Bearer $CALLAGENT_API_KEY"

Node

const registry = await fetch(`${process.env.CALLAGENT_API_URL}/registry`, {
  headers: { Authorization: `Bearer ${process.env.CALLAGENT_API_KEY}` },
}).then((response) => response.json());

GET /registry, /registry/models, /registry/model-providers, /registry/voices, /registry/transcribers and /registry/telephony — see the provider registry reference.

What is defined

These are the provider identifiers the contracts define. A provider being defined does not mean your deployment can reach it — that depends on a credential existing for the organization, and the workbench marks stages with no credential.

Stage	Providers
model	openai · anthropic · anthropic_bedrock · azure_openai · google · groq · cerebras · mistral · deepseek · xai · perplexity · together_ai · anyscale · deepinfra · openrouter · inflection_ai · minimax · custom_llm
transcriber	deepgram · assemblyai · speechmatics · gladia · soniox · azure · google · openai · openai_realtime · elevenlabs · cartesia · talkscriber · xai · custom
voice	elevenlabs · cartesia · playht · rime_ai · deepgram · openai · azure · microsoft · lmnt · hume · inworld · neuphonic · minimax · sesame · smallest_ai · wellsaid · xai · callagent · custom
speech to speech	openai_realtime · google_realtime · elevenlabs_realtime · custom_realtime
telephony	twilio · telnyx · didww · plivo · zadarma · didlogic · amazon_chime · byo_sip · custom_sip_gateway · freeswitch · web
storage	aws_s3 · cloudflare_r2 · azure_blob · gcp_cloud_storage · supabase_storage · callagent_managed

custom_llm takes any OpenAI-compatible endpoint: a baseUrl, a model, an optional credentialRef and your own headers.

Defined, registered, verified — three different things

"Defined" above means the identifier exists in the contracts. Three further questions decide whether a call will work, and they have different answers.

Defined	The identifier exists and the API will accept it.
Registered	The gateway process has an adapter that can speak the protocol.
Verified	The wire format was implemented from the vendor's current documentation and reviewed against it.

Stage	Defined	Registered	Verified
model	18	17 + a custom_llm shim	13
transcriber	14	13 + a custom shim	3
voice	19	16 + shims	2 — ElevenLabs and OpenAI
speech to speech	4	1	1

verified does not mean "tested against the vendor"

The provider SDK's own definition, verbatim:

verified: true means the wire format is implemented from the vendor's current, authoritative protocol documentation and has been reviewed against it. No adapter in this package has been run against a real endpoint in CI.

No test gates on a live credential; every test is fixture-based. Twelve voice providers have real HTTP clients pointed at real vendor URLs with an unverified wire format, and two — sesame and xai — have no vendor integration at all and only work against an endpoint you supply that speaks CallAgent's own TTS protocol. callagent as a voice provider is an indirection layer whose catalogue ships empty.

Prove your own provider with POST /credentials/{id}/test, then with a real call, before you depend on it. See What is not built.

Of the four speech-to-speech identifiers only openai_realtime has an adapter, and the speech-to-speech pipeline is not wired into the live call session at all.

The workbench

The assistant builder shows the pipeline as three cards — ear, brain, mouth — with a latency bar segmented by stage colour and an estimated cost per minute above them. Choosing a provider changes all three readings before you save anything.

Three of those readings mean different things, and the interface says which:

Reading	What it actually is
Latency	The vendor's published typical time-to-first-token or first-audio for a warm connection. CallAgent does not measure it, so it is prefixed with ~. Reading the observed p50 from your own call_metrics is not yet implemented.
Quality	An editorial 0–100 index, not a benchmark score. Also marked.
Cost	A real figure only where the registry holds a verified price. Where it does not, the card shows a relative cost band instead of a number.

A figure with a ~ is an estimate

The cost-per-minute estimate is derived from an assumed conversation shape — a fixed number of input tokens, output tokens and spoken characters per minute. It is useful for comparing two providers; it is not a quotation. The number you are actually billed comes from GET /calls/{call}/costs after the call, computed from measured quantities. See Analytics & cost.

When only some stages have a verified price, the estimate shows a partial-pricing warning rather than quietly summing what it has. A confident wrong price is worse than an obviously incomplete one, because someone will budget against it.

Bring your own key

Provider credentials are per organization and encrypted with CREDENTIAL_ENCRYPTION_KEY — deliberately not APP_KEY, so rotating the application key does not mean re-encrypting every tenant's provider secrets.

curl -X POST "$CALLAGENT_API_URL/credentials" \
  -H "Authorization: Bearer $CALLAGENT_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "class": "model",
    "provider": "openai",
    "name": "Production OpenAI",
    "secrets": { "api_key": "sk-…" }
  }'

secrets is required; the field names inside it are the vendor's own. See Quickstart step 2.

POST /credentials/{credential}/test performs a real HTTP call against the provider and records the result, so "the key is saved" and "the key works" stay separate claims. It returns 200 either way and puts the verdict in the body — read ok and verified. A provider CallAgent has no probe for comes back verified: false, which means nobody asked, not that the key is bad.

An assistant references a credential by credentialRef, never by value. The gateway is handed only the credentials one call needs, resolved at call setup — it never receives the organization's credential table.

With BYO keys the provider bills you directly and CallAgent charges only its platform fee. Both figures are still recorded, because "why is my bill this number" needs both.

Fallbacks

Each of model, transcriber, voice and speechToSpeech takes an ordered fallbacks list, maximum two entries each. Index 0 is tried first.

model:
  provider: openai
  model: gpt-4o-mini
  temperature: 0.4
fallbacks:
  model:
    - { provider: groq, model: llama-3.3-70b-versatile }
    - { provider: anthropic, model: claude-haiku-4-5 }

A fallback emits provider.fallback carrying the component, the provider moved from and to, the reason (timeout, error, rate_limit, auth, circuit_open, health_check) and the attempt number. That event is what makes "the call was slow yesterday" answerable.

Circuit breaking

A provider that keeps failing is taken out of rotation rather than retried into the ground. The breaker defaults are failureThreshold: 5, successThreshold: 2, openMs: 30000, slowCallMs: 5000, and while open the error code is circuit_open — which is retryable.

Choosing a transcriber

Endpointing quality is not the same as transcription accuracy, and for a voice agent the first matters more. A transcriber with a slightly higher word error rate but a reliable utterance-end signal produces a better-feeling call than an accurate one that leaves the assistant guessing when to speak. See Turn taking.

Keyword boosting is the other lever worth using: terms, each with a boost from -10 to 10, that bias recognition toward your product names and away from what they sound like.

It is custom_keywords, at the top level

The contract calls this transcriber.keywords, and the control plane does not read it there. The converter builds the transcriber's keyword list from the assistant's top-level custom_keywords field. Put it there.

{
  "transcriber": { "provider": "deepgram", "model": "nova-3", "smart_format": true, "numerals": true },
  "custom_keywords": [
    { "term": "Sibiu", "boost": 4 },
    { "term": "CallAgent", "boost": 6 }
  ]
}

Next

• How a call works — what each provider is responsible for.
• Assistants — where these blocks live.
• Analytics & cost — what a call really cost.