Voice agents are latency engineering dressed up as AI: the STT → LLM → TTS pipeline, the millisecond budgets, and the interruption logic that makes the loop feel human.

The pipeline: STT → LLM → TTS

A real-time voice agent is three systems pretending to be one. Speech-to-text (STT) turns the caller's audio into text, a language model (LLM) decides what to say next, and text-to-speech (TTS) turns that into audio streamed back. The hard engineering is not any one stage — it is the latency discipline and the interruption logic that make the whole loop feel like a conversation instead of a walkie-talkie.

Why WebRTC is the right transport

Voice agents shipped over plain telephony or websockets hit a wall: media quality and timing. WebRTC gives you a low-latency, bidirectional audio channel with congestion control and echo handling already built into the stack. The browser and the mobile SDKs both speak WebRTC natively, which is why every voice agent we build uses it as the transport between the user and our media servers.

// Agent session pseudo-flow
1. user audio  -> SFU      -> STT (partial results stream continuously)
2. STT result  -> LLM      -> draft response + intent
3. LLM reply   -> TTS      -> audio frames back to SFU
4. || barge-in (user speaks) -> interrupt TTS, restart STT capture

Latency budgets that feel human

People notice silence at roughly 300–500 ms. Above that, they start repeating themselves, and repetition poisons the interaction. Our working budget for a full turn is 700 ms to first audio, broken down like this:

  • STT partial result: 150–200 ms (streaming, not full-utterance)
  • LLM time-to-first-token: 150–250 ms with a small, fast model plus a routed fallback
  • TTS first frame: 100–150 ms with streaming synthesis
  • Transport and jitter allowance: the rest — and we protect it with aggressive jitter buffers

Every millisecond you save in transport is worth two in the loop, because latency compounds. If the agent is 400 ms slow today, the user adapts by talking 400 ms earlier tomorrow, and the whole rhythm drifts.

Interruption handling: the feature users feel

The moment a user says "actually," the agent has roughly a window the length of one speech segment to stop talking and start listening. We implemented barge-in at three levels:

  • Audio level: the SFU flags energy above threshold while TTS is playing.
  • STT level: a partial transcript that does not match the agent's own script aborts TTS immediately.
  • Semantic level: the LLM decides whether the interruption is a question, a correction, or noise — and whether to cut, pause, or merge.

Naive implementations only check the first level and produce agents that talk over their users. The semantic level is what converts a chatbot with a voice into a teammate.

Choosing models

We run a tiered model strategy rather than one giant model. A small STT model streams partials on cheap hardware; a medium LLM handles the conversation; a faster distilled LLM answers latency-critical turns. TTS voices are cached aggressively — the same sentence rarely needs to be synthesized twice in a session. Model choice is a latency decision first and a quality decision second; the "best" model is the one that finishes before the user gets impatient.

Where it breaks in production

  • Background noise feeds the LLM. Send audio-level VAD and speaker-turn signals downstream, or the model will argue with a truck passing by.
  • Long responses fall off the latency budget. Stream TTS sentence-by-sentence instead of waiting for the full completion.
  • Interruptions race with the token stream. Buffer the LLM output; a barge-in should discard tokens, not queue them behind TTS.
  • Retries amplify latency. A retry loop that adds 400 ms per attempt is four retries away from being unusable.

"The difference between a good agent and a great one is usually 200 milliseconds and the courage to shut up when the user starts talking."

— Sajeeb, Founder & Lead Architect, Gravity Compile

Metrics to watch, not just latency

Beyond turn latency, we track four numbers in production: barge-in rate (the fraction of sessions where the user interrupts the agent), dead-air ratio (milliseconds of silence after the user stops talking), repair rate (how often speech-to-text needs a correction), and completion rate (whether the caller finished their task). A barge-in rate above 15% usually means the agent talks too much; dead-air ratio above 400 ms means the pipeline is backlogging somewhere you cannot see. Publishing these metrics to a dashboard and reviewing them weekly changed our product decisions more than any individual model upgrade.

Testing agents in the dark

Voice agents are the hardest thing we test, because ninety percent of the failure surface is audio nobody listens to. We record and re-run sessions against the full pipeline, replay audio into a fresh speech-to-text instance during a regression suite, and keep a golden set of interrupted sentences that must produce the same barge-in decisions. It is not glamorous work, but a voice product that does not test with real, messy audio is a voice product that fails in production. Treat your recorded sessions as your most valuable dataset — they will also be the training material for every future model and prompt change.

Cost controls and model routing

Voice agents are the most expensive naive feature you can ship, because every session burns tokens in three directions at once. We treat cost as a routing problem. A routing layer decides per turn which speech-to-text model, which language-model tier and which voice to use, based on intent, language and latency pressure. Cheap intents — greetings, confirmations, small talk — never touch the expensive models; only high-value tasks escalate to the top tier.

We also cache aggressively: the same greeting synthesizes once and plays a thousand times, and common answers are pre-warmed into the speech cache at deployment. Nobody on the user's side can tell the difference between a cache hit and a fresh generation, and the monthly bill can drop by more than half.

Security and tenant isolation

Every agent deployment is multi-tenant, and we learned the hard way that prompts leak across tenants when the history buffer is shared. Sessions are isolated end to end: per-tenant keys for speech and model providers, per-tenant prompt prefixes, and per-tenant recording retention. A customer's conversation is their data — not training material, not a shared cache key. When you build voice products for regulated industries, that isolation is what lets you sleep at night.

Conclusion

Real-time voice agents are latency engineering dressed up as AI. Stream every stage, budget your milliseconds, and make interruption handling a first-class feature. We build these agents — and the infrastructure behind them — through our AI voice agent development practice, running on the same real-time stack as Gravix Cloud. If you are planning a voice product, get the latency budget right first and the models will follow.