Design an AI Voice Platform Like ElevenLabs: Text-to-Speech at Scale

The question: design an AI voice platform like ElevenLabs. Developers send text and get natural speech back, streamed fast enough for a live conversation or generated carefully for a ten-hour audiobook. Users can clone a voice from a short sample, and the system must stop people from cloning someone else's voice. It's a good interview problem because one product holds real-time streaming, heavy batch jobs, per-voice model data, usage billing and serious abuse risk.

What's public and what's our design. ElevenLabs publishes its API behaviour, model list, latency figures, regions, plan limits and safety policies, and its job postings describe Kubernetes GPU clusters for inference and training across cloud and on-prem. It hasn't published its internal serving design or model architecture. This page separates those: named figures come from the public documents (checked November 2026; they change often), and the architecture is a reasoned design for the same product, not a description of ElevenLabs' private systems.

1. What the product does

Start by naming the workloads, since they have opposite needs. A voice agent can't wait for half a second, while an audiobook can wait for an hour but must never have an audible seam.

2. Clarify requirements

3. Estimates

The natural unit here is the character, as that's how the work is requested and billed. A useful rule of thumb comes from the documented limits: 10,000 characters is about 10 minutes of speech, so 1,000 characters is about one minute. The key measurement for a GPU is the real-time factor (RTF): how many seconds of audio it produces per second of compute, including batching.

Volume       : 5B characters/day ÷ 1,000 chars per minute
               = 5M minutes of audio/day
Audio rate   : 5M ÷ 1,440 ≈ 3,470 minutes of audio per real minute
Model mix    : 70% fast model at ~50× RTF  → 2,430 ÷ 50 ≈ 49 GPUs
               30% quality model at ~10× RTF → 1,040 ÷ 10 ≈ 104 GPUs
Peak         : (49 + 104) × 3 peak factor ≈ 460 GPUs
Fleet        : 460 × 1.3 headroom ≈ 600 GPUs
Starts       : 30 s average clip → 10M clips/day ≈ 115/s, ~350/s peak
Storage      : 128 kbps MP3 ≈ 1 MB per minute
               → ~5 TB/day if all output is retained
Cost check   : 600 GPUs × ~$2/hour × 24 ≈ $29K/day
               ≈ $0.006 per 1,000 characters of compute
(illustrative numbers; ElevenLabs doesn't publish its fleet or RTF)

4. How text becomes speech

Modern neural text-to-speech systems follow a broad pattern, even though vendors don't publish details. Text is normalised and encoded. An acoustic model predicts speech features, conditioned on a voice. A vocoder (or a decoder inside an end-to-end model) turns those into a waveform. Published research such as Tacotron 2, FastSpeech 2, VITS and HiFi-GAN is the best way to learn these parts.

Streaming changes the model design. A model that needs the whole sentence before speaking sounds better, but it can't start early. A low-latency model must commit to audio with limited lookahead. That's the core trade-off between the two model families, and it's why a platform offers both instead of one best model.

5. Architecture

A worked example. A startup's assistant says *"Your order ships Friday."* using a cloned brand voice on a Flash model:

6. Preparing the text

Text models read literally, so "$3.5M" or "Dr. St. James St." can come out wrong. A normaliser turns written forms into words that can be spoken. It depends on language, context and a customer's own rules, which come from a pronunciation dictionary.

7. Streaming over WebSockets

When text is available up front, streaming over HTTP is simple. When an LLM is writing the text, you don't have it yet, so the connection must be two-way. ElevenLabs' public WebSocket API shows a clean protocol for this: send an initial message, send text as it arrives, flush at the end, and receive base64 audio along with character timing.

Buffering is a real trade-off. Speech sounds better with more context, but waiting for text adds delay. The default chunk_length_schedule of [120, 160, 250, 290] starts generating after at least 120 characters, then 160 more, and so on. If your LLM only sends 50 characters and stops, generation waits unless you flush, or use auto_mode.

Audio must be produced faster than it plays. If generation falls behind playback, the listener hears a gap (a buffer underrun). So the key real-time health metric isn't just first-byte time, it's whether each session keeps an audio buffer ahead of the playhead.

8. GPU serving and scheduling

Serving TTS is different from serving an LLM. A request produces a burst of work that lasts seconds, and audio must be stitched in order. Real-time pools run small batches with a tiny batching window. Quality pools run larger batches, or longer jobs. Use one pool per model version, so a rollout doesn't mix models within a session.

Concurrency limits protect the fleet. Rate limits per minute suit HTTP, but a generation holds a GPU slot. So plans limit simultaneous generations by model. The public limits show this clearly: heavier models get fewer slots at every plan.

Voice data must be fast to load. An instant clone is probably just a small embedding, which can be fetched in milliseconds and cached in memory. A professional clone involves heavier per-voice data, and ElevenLabs' docs note that PVC voices can be slower than default or instant voices on Flash. A sensible design keeps recent voices warm on replicas, and routes a voice to a replica that has it.

Regions. ElevenLabs lists serving from the USA, the Netherlands and Singapore, with expected time to first byte of 100-150 ms in North America, Europe and Southeast Asia, and 150-200 ms in South and Northeast Asia, on Flash over WebSockets. Enterprise customers can get EU and India data-residency environments. Data residency has its own design cost: voices and logs must stay in the region, too.

9. Long-form generation

An audiobook is too long for a single request. A 10-hour book is about 600,000 characters, while the long-form models accept 10,000 at a time. Splitting it is easy. Making the seams inaudible is the design problem.

• Conditioning on neighbours: pass the previous and next text with each paragraph. A model that knows what came before keeps the same tone, instead of restarting its intonation at every paragraph.
• Determinism and caching: hash text, voice, model version, settings and seed to get an audio key. An unchanged paragraph reuses stored audio when a user edits one line.
• Loudness: normalise each chapter to an audio target (for example, a target loudness used by audiobook platforms) so listeners don't reach for the volume control.
• Priority: background jobs run on spare capacity, with a per-account limit so that one publisher's catalogue doesn't delay everyone else.
• Pin the model version for the whole project. A model upgrade halfway through a book can change the voice, so editors should opt in.

10. Voice cloning

The public docs describe two methods, which ElevenLabs says work differently at a fundamental level. Instant Voice Cloning (IVC) works from about one to three minutes of audio and is ready almost at once. Professional Voice Cloning (PVC) trains on much more audio; third-party guides cite at least 30 minutes, with a few hours recommended, and it needs a verification step and several hours to process.

A well-known approach to instant cloning comes from research on speaker verification: train a network to tell speakers apart, and it learns an embedding that captures how someone sounds. Feed that to a TTS model, and it can speak in a voice it never trained on (the SV2TTS paper showed this in 2018).

voices(voice_id, owner_id, kind[instant|professional|library], name,
       language, embedding_ref, weights_ref, status, verification_id,
       created_at)
voice_samples(sample_id, voice_id, storage_path, duration_s,
              quality_score, consent_ref)
voice_shares(voice_id, shared_with, rate_per_1k_chars, revoked_at)

11. Safety

Cloning creates a specific harm: impersonating real people for fraud, harassment or political deception. Safety is a core design area, and it runs at three stages: before a voice is created, while speech is generated, and after audio is released. ElevenLabs' published controls include technological verification for its professional tool, blocking the cloning of celebrity and other high-risk voices, automated and human moderation, a public AI audio classifier, and support for the C2PA provenance standard.

Be honest about limits. Voice similarity can be fooled by altered audio, watermarks may not survive re-recording, and a classifier has false positives. Say that layered controls raise the cost of abuse without removing it, and that a human team and a fast takedown process stay necessary.

12. Voice agents on the platform

A voice agent is a major customer of these real-time APIs, and voice platforms increasingly offer it as a product. ElevenLabs' docs list real-time speech recognition (scribe_v2_realtime, about 150 ms) and eleven_realtime_v1 models for agents, alongside its Agents Platform. Turn detection, barge-in and tool calls are covered in depth on the voice AI agent page.

13. Usage, billing and caching

Character billing looks simple, but it has edge cases. If a stream fails halfway, charge only for audio produced. If a user cancels a barge-in, charge for what was generated. Retries with the same request id mustn't charge twice. Treat metering as an event stream with idempotent keys, not a counter updated on the request path. See idempotency.

Caching audio pays off for repeated phrases such as "Please hold" and app prompts. The key must include everything that changes the sound: normalised text, voice, model version, settings, language and format. Cache only for customers who allow retention, and remember that a voice's owner may later revoke it, so the cache must honour that.

14. Quality and operations

Audio quality is hard to score with a formula. Combine automated checks with human listening, and test on the things that break most: numbers, names, unusual languages and long passages.

A clever check: run generated speech back through speech recognition and compare it with the original text. A mismatch on "$3.5M" flags a normalisation bug, with no human needed. It won't catch poor emotion or tone, which still needs listeners.

Follow-up questions to expect

• *How would you support emotional delivery tags, like [whispers]?* Extend the text encoder to treat tags as control tokens, then evaluate them with listening tests. Moderate tags as well, and strip them from captions.
• *A voice owner removes their consent. What happens?* Mark the voice as revoked so that new generations fail, purge caches, delete embeddings and weights, and respect any existing contracts on delivered audio.
• *How do you share voices in a marketplace?* Keep the owner's rights and payouts in a separate table, meter usage per voice, and let the owner revoke it. Never expose the sample audio or embedding to the buyer.
• *Dubbing into another language?* Transcribe, translate with timing constraints, generate speech in the same voice, and align it to the video. It's a pipeline over this platform.
• *Real-time customers report a slow first byte. Where do you look?* Trace it as below.