Design a Real-Time Voice AI Agent: Speech, LLM and TTS Latency

System design for a real-time voice AI agent: streaming speech recognition, turn detection, LLM and tools, streaming text-to-speech, interruptions, latency.

Advanced 3 min read · Updated 29 October 2026

The question: design a real-time voice AI agent that answers phone calls or in-app voice conversations, for example to book appointments or handle support. It must listen, understand, think, use tools and speak back with a delay short enough to feel like a conversation, and it must handle people interrupting it. Voice turns every latency problem in AI engineering into something the user can hear.

1. Clarify requirements

Question	Answer
Channel?	Phone (telephony) and in-app (WebRTC)
Tasks?	Answer FAQs, look up and book appointments via tools, transfer to a human
Scale?	5,000 concurrent calls at peak
Latency?	Agent starts speaking within ~800 ms of the user finishing, most of the time
Behaviour?	Users can interrupt (barge-in); the agent stops and listens
Compliance?	Disclose that it's an AI; recording consent; PII handling

2. Two architectures

	Cascaded pipeline	Speech-to-speech model
How	Speech recognition (ASR) → LLM → text-to-speech (TTS)	One model takes audio in and produces audio out
Latency	Sum of stages; needs streaming everywhere	Potentially lower
Control	Text in the middle: easy to log, guard, evaluate and use tools	Harder to inspect and constrain
Flexibility	Swap any component; pick best ASR/LLM/TTS per language	Tied to one model's capabilities

Most production agents use a streaming cascade, because the text stage makes tools, guardrails and evaluation straightforward. The design below uses it.

3. Architecture

A streaming voice pipeline. Audio streams in over WebRTC or a telephony bridge; streaming ASR produces partial transcripts; turn detection decides when the user has finished; the LLM streams text into streaming TTS; audio streams back. Barge-in cancels speech immediately.

4. The latency budget

Time from the user's last word to the agent's first audio, with every stage streaming (illustrative). Turn detection and the LLM's time to first token are the biggest items.

Stream everything: ASR emits partial text while the user speaks; the LLM streams tokens; TTS starts speaking from the first phrase, not the full answer.
Turn detection: waiting for a fixed silence (say 700 ms) is slow and cuts off people who pause. Combine voice activity detection with a small model that predicts end-of-turn from the transcript.
Fast models: a small or mid-size LLM tuned for short spoken answers; keep prompts compact and use prefix caching.
Co-locate ASR, LLM and TTS in the same region as the media server to avoid extra network hops.
Fillers for tool calls: if a lookup takes a second, say "Let me check that" while it runs.

5. Interruptions (barge-in)

While the agent speaks, keep listening. When voice activity indicates the user is speaking (and it's not echo of the agent's own audio), stop TTS immediately and cancel the in-flight LLM generation.
Record what was actually spoken before the interruption, so conversation history matches what the user heard.
Use echo cancellation so the agent doesn't interrupt itself.

6. Conversation design for voice

Short sentences, one question at a time, no lists or markdown; confirm critical details ("That's Tuesday at 3 pm, correct?").
Spell out numbers and dates properly for TTS; handle ASR errors on names and numbers by confirming.
Tools work as in any agent: typed, permission-checked, with idempotent booking calls.
Transfer to a human with a summary when stuck or asked.

7. Scale and cost

Concurrency : 5,000 calls; each holds a media session plus streaming ASR and TTS
ASR         : one streaming session per active call → 5,000 concurrent streams
LLM         : a turn every ~10 s per call → ~500 LLM requests/s, short outputs
TTS         : speaking ~40% of call time → ~2,000 concurrent synthesis streams
Media       : stateful sessions pinned to servers; drain before deploys
Cost/minute : ASR + LLM + TTS + telephony per minute of call; compare with the
              cost of a human-handled minute

8. Evaluation and operations

Latency percentiles per stage and end to end; interruption handling success; ASR word error rate on names and numbers.
Task success (bookings made correctly), transfer rate, call duration, caller satisfaction.
Simulated calls with synthetic voices and noise for regression testing; sampled call review. See LLM evaluation.
Recordings and transcripts are sensitive: consent, redaction and retention policies.

Where it’s used

Call centres

Appointment booking, order status and triage before human transfer.

In-app voice assistants

Hands-free help in driving, cooking or fitness apps over WebRTC.

Language practice and tutoring

Conversational practice where natural turn-taking matters most.

In an interview

Compare a streaming cascade with speech-to-speech and justify the choice.
Build a latency budget per stage and stream every stage.
Design turn detection and barge-in with cancellation.
Cover voice-specific conversation design, tools and evaluation.

Frequently asked questions

How do you build a low-latency voice AI agent?

Stream every stage: streaming speech recognition, fast end-of-turn detection, an LLM streaming its first sentence quickly, and streaming text-to-speech, all co-located near the media server, with barge-in that stops speech when the user talks.

Cascaded pipeline or speech-to-speech model?

A streaming cascade of speech recognition, LLM and text-to-speech is easier to inspect, guard, evaluate and connect to tools, so most production agents use it; speech-to-speech models can be faster but are harder to control.