Sign inStart practicing
Start practicing Sign in

Generative AI and LLM System Design Interview Questions (2026)

LLM, RAG and AI agent system design interview questions asked of AI engineers, grouped by theme, with what interviewers probe and a worked answer for each.

Beginner 3 min read · Updated

These are the generative AI and LLM system design interview questions that AI engineer, LLM engineer and applied AI candidates are asked most, grouped by theme. They come up at model providers, big tech AI teams and the many startups building on LLMs. For each one you'll see what interviewers probe and where to study it. Answer them with the AI engineer interview framework.

Across company reports, three prompts dominate: a production RAG system, a tool-using agent (often customer support), and LLM inference or batching. Prepare those three deeply before anything else.

The themes at a glance

Doc chatbot, answer engineRAG and searchretrieve → generateSupport, workflow agentsAgentstools · loopsChatGPT, copilotsAssistantschat at scaleBatching, gatewaysServingGPUs · tokensText-to-image, voiceGenerationimage · voiceEval harness, moderationQuality + safetyevals · guardsGenAI design questions
Generative AI design questions cluster into six themes. Each theme shares a core architecture and a set of follow-ups.

RAG, search and knowledge

QuestionWhat interviewers probeStudy
Design a RAG chatbot over company documentsPermissions, freshness, chunking, hybrid search, citations, evaluationWorked answer · Round
Design an AI answer engine like PerplexityQuery planning, web retrieval, source ranking, grounding, latencyWorked answer · Round
Design a document processing pipeline for LLM useParsing PDFs and scans, chunking at scale, incremental re-indexing, deletesWorked answer · Round
Debug a RAG system that answers wrongly when the document existsSeparating retrieval failures from generation failuresAdvanced RAG
Handle context window limits in a document-heavy systemChunk selection, summarisation, long context vs RAGChunking strategies

Agents and tool use

QuestionWhat interviewers probeStudy
Design a customer support AI agentTool contracts, approvals, human handoff, evals, token costWorked answer · Round
Design an enterprise tool-using agentLeast-privilege credentials, state, long-running tasks, auditAI agent architecture
Design an insurance-claims agent using RAG under a token budgetPer-task budgets, routing risky cases to humans, citationsCost optimization
Design a scalable, safe multi-agent systemOrchestration, trust boundaries, loops, concurrencyMulti-agent systems
Agent or deterministic workflow?Judgment: when open-ended planning is actually neededAI agent architecture
Connect an agent to many internal toolsFunction calling, MCP servers, auth, tool discoveryTool calling and MCP

Assistants and copilots

QuestionWhat interviewers probeStudy
Design ChatGPTConversation storage, streaming, context management, GPU capacity, safetyWorked answer · Round
Design GitHub CopilotCompletion latency, repository context, privacy, acceptance metricsWorked answer · Round
Design Gmail Smart ComposeTiny latency budgets, small models, personalisation, safety filtersModel compression
Design a chatbot with long-term memoryWhat to remember, retrieval of memories, privacy, poisoningAgent memory
Design a file uploader that feeds large multimodal files into model contextChunked upload, parsing, context budgetingResumable uploads

LLM serving and infrastructure

QuestionWhat interviewers probeStudy
Design an inference batching system for LLM queriesDynamic vs continuous batching, throughput vs latency, backpressureLLM serving · Round
Design an API for sampling from LLMs efficientlyStreaming, KV cache reuse, rate limits, routingLLM serving
Design an LLM gateway for a companyProvider fallback, budgets, keys, caching, observabilityLLM gateway · Round
Design a GPU credit / quota systemMetering tokens, quotas, billing correctnessRate limiting
Design an LLM playgroundSessions, model selection, streaming, cost display, abuseDesign ChatGPT
Design a low-latency model serving platformAutoscaling GPUs, canaries, multi-model hostingRound

Multimodal generation

QuestionWhat interviewers probeStudy
Design a text-to-image service like MidjourneyGPU job queues, priorities, safety on prompts and images, storageWorked answer · Round
Design a real-time voice AI agentStreaming ASR and TTS, turn detection, interruptions, latency budgetWorked answer · Round
Design image captioning or video summarisationBatch pipelines, multimodal models, evaluationLLM evaluation

Evaluation, safety and operations

QuestionWhat interviewers probeStudy
Design an evaluation harness for an agentRepeatable trials, traces, graders, regression gatesLLM evaluation · Round
Can you trust LLM-as-a-judge?Calibration with humans, position and verbosity biasLLM evaluation
Secure a RAG system or tool-using agentIndirect prompt injection, data exfiltration, authorisationPrompt injection
Design guardrails for a public chatbotInput and output classifiers, PII, refusals, fallbacksLLM guardrails
Respond to an incident after a prompt or model changeVersioning, rollback, reproduction, canariesLLMOps
Cut LLM costs by half without hurting qualityCost per task, routing, caching, context, batchingCost optimization

How to prepare

  1. Work through the AI engineer roadmap stages 1 to 8; each maps to a group of follow-ups above.
  2. Study three worked answers closely: RAG chatbot, support agent, ChatGPT.
  3. For every design, practise saying the eval plan, the token estimate and the top three risks in under two minutes.
  4. Do timed rounds out loud with follow-ups; the RAG practice round is a good first one.

Frequently asked questions

What generative AI system design questions are asked in interviews?

Common ones are design a RAG chatbot over company documents, a customer support AI agent, ChatGPT, GitHub Copilot, an LLM inference batching system, an LLM gateway, a text-to-image service, a voice AI agent, and evaluation or guardrail systems.

Which GenAI design question should I prepare first?

A production RAG system, then a tool-using agent, then LLM serving and batching. Together they cover retrieval, permissions, evaluation, tools, safety, latency and cost.

Further reading