The brief
Serve a chat assistant to millions of people with streamed answers, saved history and fair usage limits. Your design should meet these requirements:
- Tens of millions of people a day hold multi-turn conversations; answers stream token by token and the first words appear within about a second.
- Conversations are saved, can be reopened and continued, and long conversations still work.
- Free and paid plans get different models and usage limits, and harmful requests and answers are caught.
Scale to design for
A consumer chat assistant with about 50 million daily users sending around 10 messages each: roughly 500 million messages a day, about 6,000 a second on average and 15,000 at peak. A typical request carries about 2,500 input tokens (system prompt, history, message) and produces about 400 output tokens. The first token should appear within about a second; answers stream. The company runs its own models on a GPU fleet. Free and paid plans differ in model and limits.
State these assumptions out loud in the interview, then let them drive your choices: they decide what needs caching, sharding or asynchronous processing.
Before you start
You'll get more out of this round if you're comfortable with:
- Tokens and context windows
- LLM inference: prefill, decode and batching
- Streaming over SSE or WebSockets
- Rate limiting
Questions an interviewer may ask
Expect follow-ups like these once your high-level design is on the board. Have an answer, and a reason, for each:
- How many GPUs do you need, and what drives that number?
- A conversation is 200 turns long: what do you send to the model?
- The user presses stop halfway through an answer: what happens on the server?
- Traffic doubles during a launch and GPUs are full: who waits and who gets a smaller model?
- How do you store conversations so regenerate and edits work?
Common mistakes to avoid
- Sizing by requests per second instead of tokens
- Resending entire unbounded histories
- No cancellation so stopped answers keep burning GPUs
- One global queue with no priorities
- Storing conversations in a way that can't list a user's chats quickly
- Treating rate limits as requests instead of tokens
- No plan when GPUs are saturated
Concepts this round covers
- Token-based capacity planning
- Streaming and cancellation
- Context management
- Assistants
What a complete answer covers
Estimates in tokens, a conversation store, context assembly for long chats, a streaming path from a GPU inference fleet with cancellation, plan-based limits and overload behaviour, moderation, and how quality and cost are measured.
There is no single correct design. A strong answer meets the requirements at the stated scale and explains the trade-offs behind each choice. New to the format? Read how to prepare for a system design interview.