AI Engineer Roadmap 2026: LLM Apps, RAG, Agents and Evals
How to become an AI engineer, step by step: LLM basics, prompting, RAG, agents and tool calling, evals, guardrails, LLMOps, cost and system design interviews.
This AI engineer roadmap shows how to become an AI engineer from a system design point of view: how to build reliable products on top of large language models (LLMs) and other foundation models, and how to answer the LLM, RAG and AI agent system design questions that AI engineering interviews now ask. It goes from how LLMs work and how to choose one, through prompting and structured output, retrieval-augmented generation, agents and tool calling, evaluation, guardrails and prompt injection, serving, gateways, caching and cost, fine-tuning, and LLMOps, and ends with full worked answers to the most-asked generative AI design questions. Each step links to a free guide.
The step most candidates skip is evaluation. Saying how you'll measure quality before you design (a golden dataset, graders, a regression gate) is the clearest signal of an experienced AI engineer.
Builds on the core system design roadmap. Do that first, or skip the parts you already know; this one only covers what ai engineering interviews go deeper on.
Step 1The AI engineering design round
How LLM system design rounds differ from backend and ML ones, and the framework to answer them.
Step 2How LLMs work
Tokens, context windows, sampling and pricing, and how to pick a model for a job.
Step 3Prompting and structured output
Prompts as versioned code, and model output your services can trust.
Step 4Retrieval-augmented generation
Grounding answers in your own data: ingest, chunk, retrieve, rerank, cite.
- RAG architectureIntermediate
- RAG chunking strategiesIntermediate
- Vector databasesIntermediate
- Hybrid search and rerankingIntermediate
- Advanced RAGAdvanced
- Search and inverted indexGood to know
Practice:Retrieval-augmented generation systemAI engineering · AdvancedDocument ingestion pipeline for RAGAI engineering · AdvancedAI answer engineAI engineering · Advanced
Step 5Agents and tools
Letting a model take actions: the agent loop, tool contracts, memory and multi-agent designs.
- AI agent architectureIntermediate
- Tool calling and MCPIntermediate
- Agent memoryIntermediate
- Multi-agent systemsAdvanced
- IdempotencyGood to know
- SagasGood to know
Practice:Customer support AI agentAI engineering · Advanced · FreeReal-time voice AI agentAI engineering · Advanced
Step 6Evaluating LLM systems
Golden datasets, graders and regression gates: the step most designs skip.
Practice:LLM evaluation platformAI engineering · Intermediate
Step 7Safety and security
Guardrails, prompt injection and least-privilege tools.
Step 8Serving, latency and cost
Running models fast and affordably: inference, gateways, caching and routing.
- LLM servingAdvanced
- LLM gatewayIntermediate
- LLM cachingIntermediate
- LLM cost optimizationIntermediate
- Model compressionGood to know
- FallbacksGood to know
Practice:LLM gatewayAI engineering · IntermediateLLM inference platformAI engineering · AdvancedText-to-image generation serviceAI engineering · Advanced
Step 9Adapting models
When prompting and RAG aren't enough: fine-tuning, preference tuning and distillation.
Step 10LLMOps in production
Tracing, versioning, releases, feedback and incident response for AI features.
Step 11Practise GenAI designs
Full worked answers to the AI engineering questions asked most, then timed practice.
- Design ChatGPTAdvanced
- Design a RAG chatbotAdvanced
- Design a support AI agentAdvanced
- Design an AI code assistantAdvanced
- Design a voice AI agentGood to know
- Design text-to-imageGood to know
- Design document ingestionGood to know
- Design an AI answer engineGood to know
Practice:ChatGPTAI engineering · Advanced · FreeCustomer support AI agentAI engineering · Advanced · FreeAI code assistantAI engineering · AdvancedReal-time voice AI agentAI engineering · AdvancedText-to-image generation serviceAI engineering · AdvancedDocument ingestion pipeline for RAGAI engineering · AdvancedAI answer engineAI engineering · AdvancedRetrieval-augmented generation systemAI engineering · Advanced
What an AI engineer does
An AI engineer builds applications on foundation models that someone else trained: assistants, search and question answering, agents that take actions, extraction pipelines, coding tools. It differs from an ML engineer, who trains task-specific models from labelled data (see the ML system design roadmap). The two overlap in evaluation, serving and monitoring.
| ML engineer | AI engineer | |
|---|---|---|
| Starting point | Data and labels; train a model | A pretrained model; adapt it with prompts, context and tools |
| Main levers | Features, model architecture, training data | Prompts, retrieval, tool design, model choice, fine-tuning |
| Evaluation | Offline metrics on labelled test sets | Golden datasets, LLM-as-a-judge, human review, online feedback |
| Cost driver | Training compute, feature pipelines | Tokens per request, GPU inference, agent loops |
| Typical failures | Drift, skew, feedback loops | Hallucination, prompt injection, runaway agents, latency |
The shape of an LLM application
The AI engineering design interview framework
| Step | Questions to answer | Guide |
|---|---|---|
| 1. Clarify | Users, task, quality bar, cost of a wrong answer, latency, volume, data sensitivity | AI engineer interview |
| 2. Define evals first | Golden set, graders, success and guardrail metrics | LLM evaluation |
| 3. Pick the approach | Prompting, RAG, tools or agent, fine-tuning: the least complex that works | Fine-tuning vs RAG |
| 4. Design the context | Retrieval, chunking, memory, prompt assembly | RAG, agent memory |
| 5. Orchestration and tools | Workflow vs agent, tool contracts, approvals, stop conditions | Agents, tool calling and MCP |
| 6. Safety | Guardrails, prompt injection, permissions | Guardrails, prompt injection |
| 7. Serve and scale | Model routing, streaming, caching, capacity in tokens, cost per task | LLM serving, gateway, cost |
| 8. Operate | Tracing, versioning, canaries, feedback, incidents | LLMOps |
Core tools by category
| Category | Common examples | Guide |
|---|---|---|
| Model APIs | Claude, GPT, Gemini, Mistral and other hosted models | Choosing an LLM |
| Open-weight models and serving | Llama, Qwen, Mistral families; vLLM, TGI, TensorRT-LLM | LLM serving |
| Vector search | pgvector, FAISS, dedicated vector databases, search engines with vectors | Vector databases |
| Orchestration frameworks | LangChain, LlamaIndex, provider agent SDKs, or plain code | AI agent architecture |
| Tool integration | Function calling, Model Context Protocol (MCP) servers | Tool calling and MCP |
| Evaluation and tracing | Eval harnesses, LLM-as-a-judge, tracing platforms, OpenTelemetry | LLM evaluation, LLMOps |
Frameworks change monthly. Interviews test whether you understand what each layer does, so you can build it with any tool, or without one.
A six-week plan
| Week | Stages | Practise |
|---|---|---|
| 1 | 1-3 | Learn the framework; estimate tokens and cost for a chat feature; write a versioned prompt with a JSON schema output and validation. |
| 2 | 4 | Design a RAG system over company documents: chunking, hybrid search, reranking, citations, permissions. Then the RAG round. |
| 3 | 5 | Design a customer support agent with three tools, approval for refunds, stop conditions and a human handoff. |
| 4 | 6-7 | Build a golden set and grading plan for the support agent; list prompt injection paths and defences for it. |
| 5 | 8-10 | Add a gateway with routing and fallbacks, caching and per-tenant budgets; plan tracing, canary releases and an incident runbook. Try the model serving round. |
| 6 | 11 | Timed mocks from the GenAI question list, starting with design ChatGPT and the RAG chatbot. |
Skills checklist
- Explain tokens, context windows, sampling and how they drive cost and latency.
- Choose between prompting, RAG, an agent and fine-tuning, and defend the choice.
- Design a RAG pipeline end to end, including permissions, freshness and evaluation.
- Design an agent with typed tools, budgets, stop conditions and human approval.
- Build an eval set with code checks and calibrated LLM judges, and gate releases on it.
- Defend against direct and indirect prompt injection with least privilege.
- Estimate GPU or API cost per task and cut it with routing, caching and context trimming.
- Trace, version and roll back prompts, models and retrieval configs.
Frequently asked questions
How do I become an AI engineer?
Learn how LLMs work and how to choose one, then prompting and structured output, retrieval-augmented generation, agents and tool calling, evaluation, guardrails and prompt injection, serving, caching and cost, fine-tuning, and LLMOps. Build small projects for each and practise designing full systems such as a RAG chatbot or a support agent.
What is the difference between an AI engineer and an ML engineer?
An ML engineer usually trains task-specific models from labelled data and runs them in production. An AI engineer builds applications on pretrained foundation models, adapting them with prompts, retrieved context, tools and sometimes fine-tuning. Both need evaluation, serving and monitoring skills.
What system design questions do AI engineers get asked?
The most common are designing a production RAG system over company documents, a customer support agent that uses tools, an LLM inference or batching system, ChatGPT, a coding assistant, and evaluation or guardrail systems for LLM applications.
Further reading
- AI Engineering: Building Applications with Foundation ModelsChip Huyen, O'Reilly
- Building effective agentsAnthropic
- OWASP Top 10 for LLM ApplicationsOWASP GenAI Security Project