Design ChatGPT: LLM Chat Assistant System Design Walkthrough

The question: design ChatGPT, a consumer chat assistant where millions of users hold multi-turn conversations with a large language model, see answers stream in token by token, return to old conversations, and use features like file uploads and memory. The interesting parts are not the chat UI: they're context management, streaming at scale, GPU capacity and cost, rate limits by plan, and safety. This walkthrough follows the AI engineer interview framework.

1. Clarify requirements

2. Estimates

Messages      : 50M × 10 = 500M/day ≈ 5,800/s avg, ~15K/s peak
Tokens/message: input ≈ 2,500 (system + history + message), output ≈ 400
Daily tokens  : 500M × 2,500 = 1.25T input; 500M × 400 = 200B output
Decode load   : 200B ÷ 86,400 ≈ 2.3M output tokens/s avg, ~6M/s peak
GPU fleet     : if one GPU server sustains ~5K output tokens/s with batching,
                peak needs ~1,200 servers for decode, plus prefill capacity and headroom
Storage       : 500M messages × ~2 KB = 1 TB/day of conversation text
Concurrency   : 15K msgs/s × ~10 s streaming = ~150K open streams at peak
(illustrative throughput; real numbers depend on model size and hardware)

Conclusions: GPU inference dominates cost, output tokens drive capacity, and the system must hold a very large number of long-lived streaming connections. See how LLMs work.

3. High-level architecture

4. Conversation storage

• Data model: conversations(id, user_id, title, model, created_at, updated_at) and messages(conversation_id, seq, role, content, parent_id, tokens, created_at). parent_id supports regenerate and edits as branches.
• Access patterns: list a user's recent conversations; load one conversation's messages in order; append messages.
• A wide-column or document store partitioned by user_id (conversations) and conversation_id (messages) fits; see data modeling.
• Writes: the user message is stored before generation; the assistant message is written when streaming completes (with partial saves for long answers, so a dropped connection doesn't lose them).

5. Context management

Each request resends what the model needs: system prompt, relevant memory, conversation history and the new message. Long conversations exceed the context window and cost more per turn, so:

• Keep recent turns verbatim, summarise older ones, and drop or summarise large pasted content after it's been answered.
• Put the stable system prompt first to benefit from prefix caching; see LLM caching.
• Long-term memory ("remember I'm vegetarian") is retrieved per turn from a per-user store; see agent memory.
• Count tokens before calling the model and enforce plan-specific limits.

6. Streaming responses

• Server-Sent Events are a natural fit (one-way, HTTP, reconnectable); WebSockets also work. See real-time communication.
• Stop must cancel generation on the GPU, freeing capacity, not just stop sending.
• If the client disconnects, finish or cancel generation and save what exists; the client can fetch it on reconnect.

7. Inference fleet and scheduling

• Serving engines with continuous batching and a paged KV cache maximise GPU throughput; see LLM serving.
• A router picks the model by plan and request, then a replica with free KV cache memory; queues per model with priority for paid users.
• Under overload: queue briefly, then degrade (smaller model for free users, shorter max output) rather than fail; see load shedding.
• Autoscale on queue depth and KV cache utilisation, but GPU capacity is slow to add, so keep headroom and plan for daily peaks.

8. Rate limits and plans

Limit by tokens and messages per time window per user and plan, enforced at the gateway with a shared counter store. Free users get a smaller default model and lower caps; paid users higher caps and priority. Abuse detection (scripts, shared accounts) feeds into stricter limits. See rate limiting and LLM gateway.

9. Safety

• Input moderation before generation; output moderation on streamed chunks, with the ability to cut a stream.
• Policy-tuned refusals; special handling for self-harm topics (resources, not refusal).
• File uploads scanned and parsed in isolated workers; uploaded content treated as untrusted (see prompt injection).
• See LLM guardrails.

10. Evaluation and operations

• Offline eval suites for each model and system prompt version; red-team sets for safety.
• Online: thumbs up/down, regenerate rate, conversation length, retention by cohort; A/B tests for model and prompt changes.
• Dashboards: time to first token, tokens per second per stream, GPU utilisation, queue wait, cost per message.
• See LLM evaluation and LLMOps.

Follow-up questions to expect

• *How do you generate conversation titles?* Asynchronously, with a small model, after the first exchange.
• *How would you add web search?* As a tool the model can call, with retrieved pages treated as untrusted; see AI answer engine.
• *How do you handle a GPU region outage?* Route to other regions with capacity, degrade free tier first, and keep conversation storage multi-region.