AI Engineer System Design Interview: A Framework for LLM Systems
How to answer AI engineer and LLM system design interview questions: a framework covering quality, context, agents, evals, guardrails, cost and latency.
An AI engineer system design interview asks you to design a product built on large language models: a chat assistant, a RAG chatbot over company documents, a customer support agent, a coding assistant, a document processing pipeline. Interviewers look for the same judgment as any design round, plus things specific to LLMs: how you'll measure quality, where the model's context comes from, how agents and tools are kept safe, and how cost and latency stay under control. This guide gives an eight-step framework, an estimation method in tokens, and the follow-ups to expect.
Start simple and escalate: a single prompt, then retrieval, then tools, then an agent, then fine-tuning. Interviewers reward the candidate who reaches for an agent only when a fixed workflow can't do the job.
How these rounds differ
| Topic | Backend round | ML round | AI engineering round |
|---|---|---|---|
| Core risk | Data loss, downtime | Model silently wrong | Confidently wrong answers, unsafe actions, runaway cost |
| Quality proof | Correctness by design | Offline + A/B metrics | Golden datasets, graders, human review, online feedback |
| Capacity unit | Requests per second | Predictions per second | Tokens per second and GPU memory |
| Special components | Caches, queues, DBs | Features, training, serving | Retrieval, prompts, tools, guardrails, gateway, evals |
The eight-step framework
1. Clarify the task and the stakes
Who uses it and for what? What does a wrong answer cost (a bad movie suggestion vs a wrong refund vs wrong medical advice)? What data can the model see, and who is allowed to see what? Latency expectations (streaming chat vs a 10-second report)? Volume in requests and in tokens?
2. Define success and evaluation first
Name the quality dimensions (correctness, groundedness, helpfulness, format, safety), how each is graded (code checks, an LLM judge calibrated against humans, human review), and the release gate. See LLM evaluation.
3. Choose the simplest approach that can work
| Approach | Use when | Guide |
|---|---|---|
| Single prompt | The model already knows enough; task is self-contained | Prompt engineering |
| Prompt + structured output | Results feed code: extraction, classification, routing | Structured outputs |
| RAG | Answers depend on private or fresh data | RAG |
| Workflow with tools | Known steps that call systems (look up order, then summarise) | Tool calling |
| Agent | Steps can't be known in advance; open-ended tasks | AI agents |
| Fine-tuning | Style, format or narrow skill the above can't reach, or to cut cost with a smaller model | Fine-tuning vs RAG |
4. Design the context
What goes into each prompt: system instructions, retrieved chunks, conversation history or memory, tool results. How retrieval works (chunking, hybrid search, reranking), how history is trimmed or summarised, and how permissions filter what's retrieved. See chunking, hybrid search and agent memory.
5. Orchestration and tools
Typed tool schemas, validation of arguments, idempotency for side effects, budgets on steps and tokens, stop conditions, and human approval for risky actions. Say whether one agent or several, and why. See multi-agent systems.
6. Safety and security
Input and output guardrails, treating retrieved and tool content as untrusted, least-privilege credentials, and authorisation checks in code, not in prompts. See guardrails and prompt injection.
7. Serving, latency and cost
Hosted API or self-hosted model, streaming, model routing (small model for easy requests), caching, a gateway with fallbacks and budgets, and capacity in tokens. See LLM serving, LLM gateway, caching and cost optimization.
8. Operate
Trace every model and tool call, version prompts and configs, run evals in CI, release behind canaries, collect user feedback, and have a rollback and incident plan. See LLMOps.
Estimating in tokens
LLM capacity and cost scale with tokens, not requests. Do this arithmetic out loud:
Assistant for 2M daily users, 6 messages each = 12M requests/day ≈ 140 req/s avg, ~400 peak
Per request: input ≈ 1,500 system+history + 1,500 retrieved = 3,000 tokens
output ≈ 300 tokens
Daily tokens: 12M × 3,000 = 36B input, 12M × 300 = 3.6B output
Cost (illustrative prices: $1 per 1M input, $5 per 1M output for a mid model):
36,000 × $1 + 3,600 × $5 = $36K + $18K = $54K/day
Levers: prompt caching on the 1,500-token system prefix, routing 60% of
traffic to a model ~5× cheaper, trimming retrieved context to 800 tokens
Latency: time to first token ~0.3-1 s; 300 output tokens at ~50 tokens/s ≈ 6 s
→ stream the response; the user sees text in under a secondReplace the prices with whatever the interviewer gives you; the method is what's assessed. See how LLMs work for what drives these numbers.
A 45-minute time plan
Follow-up questions to expect
- *The answer is wrong but the document exists. How do you debug it?* Check ingestion and freshness, permission filters, chunk boundaries, query rewriting, retrieval recall, reranking, context assembly, then generation, measuring retrieval separately from generation.
- *Quality is good but it's too slow and expensive.* Measure per-stage latency and cost per resolved task; route to smaller models, trim context, cache, parallelise tool calls, stream, cut agent turns.
- *A prompt change caused an incident.* Roll back by version, keep traces, reproduce with the exact configuration, add the case to the eval set, release again behind a canary with abort thresholds.
- *How do you stop the agent doing something harmful?* Least-privilege tools, argument validation, approval for irreversible actions, budgets and stop conditions, and checking outcomes in the system of record.
- *Can you trust an LLM judge?* Only after calibrating it against human labels, controlling for position and length bias, and re-checking when the judge or rubric changes.
Common mistakes
- Jumping to a multi-agent architecture for a task a two-step workflow handles.
- No evaluation plan, or "we'll check a few examples manually".
- Relying on the system prompt for authorisation ("don't show other users' data").
- Ignoring token costs and context limits until the end.
- Treating the model as deterministic: no retries, validation or fallbacks for malformed output.
Where it’s used
Usually one LLM system design round, plus coding and a project deep dive; RAG and agent designs are the most common prompts.
More serving depth: batching for inference, sampling APIs, GPU capacity and cost.
A backend design round with an LLM feature in it; evals, guardrails and cost usually decide the outcome.
In an interview
- Ask what a wrong answer costs before choosing an approach.
- Define evaluation in the first ten minutes.
- Escalate from prompt to RAG to tools to agent only as needed, and say why.
- Estimate tokens, cost and latency, then name the levers that cut them.
- Put authorisation in code and treat retrieved and tool content as untrusted.
Frequently asked questions
How do I answer an LLM system design question?
Clarify the task and the cost of a wrong answer, define evaluation first, pick the simplest approach (prompting, RAG, tools, agent, fine-tuning), design the context, design tools with safety controls, add guardrails, estimate tokens, latency and cost, and plan tracing, versioning and rollback.
How is an AI engineering design round different from a backend one?
It adds quality proof through evaluation, context design through retrieval and memory, safety against prompt injection and unsafe actions, and capacity measured in tokens and GPU memory rather than requests alone.
Further reading
- AI Engineering: Building Applications with Foundation ModelsChip Huyen, O'Reilly
- Building effective agentsAnthropic