Sign inStart practicing
Start practicing Sign in

ML System Design Roadmap: From Data to Production Models

An ML system design roadmap: features, training, serving, ranking and recommendations, evaluation, monitoring, plus RAG, vector search and LLM serving.

Intermediate 3 min read · Updated

This ML system design roadmap covers what machine learning interviews ask beyond modelling: how features are computed consistently for training and serving, how models are trained at scale and versioned, how predictions are served within a latency budget, how recommendation and ranking systems are structured, how you prove a model helps, and how you notice when it stops helping. The last step covers AI applications built on foundation models: retrieval-augmented generation, vector search and LLM serving.

ML design rounds reward candidates who talk about data and feedback loops as much as models. If you can only spend time on two steps, make them step 2 (features) and step 6 (evaluation and monitoring).

Builds on the core system design roadmap. Do that first, or skip the parts you already know; this one only covers what machine learning systems interviews go deeper on.

Tick topics off as you read them

  1. Step 1The end-to-end ML system

    How data, training, serving and feedback connect.

  2. Step 2Data and features

    Consistent features for training and serving.

    Practice:Feature storeML & AI · Advanced

  3. Step 3Training at scale

    Turning data into models reproducibly.

    Practice:Distributed model training platformML & AI · Advanced

  4. Step 4Serving predictions

    Meeting latency and cost budgets in production.

    Practice:Low-latency ML model serving platformML & AI · Advanced

  5. Step 5Ranking and recommendations

    The most common ML design question.

    Practice:Recommendation systemML & AI · Advanced · Free

  6. Step 6Measuring and monitoring

    Knowing whether a model actually helps.

    Practice:ML model monitoring systemML & AI · IntermediateReal-time fraud scoringML & AI · Advanced

  7. Step 7LLM and AI applications

    Building products on foundation models.

    Practice:Retrieval-augmented generation systemML & AI · Advanced

The ML system design framework

PhaseQuestions to answerGuide
Problem framingWhat business metric moves? What does the model predict, and what's the label?ML evaluation
Data and featuresWhere do labels come from? Which features exist at prediction time?Feature stores
Model and trainingWhat's the simplest baseline? How often is it retrained, on what data?Distributed training, MLOps
ServingBatch or online? What's the latency budget? What happens when the model is down?Model serving
EvaluationWhich offline metric, which online metric, which guardrails?A/B testing
MonitoringHow do you detect drift and broken features before users do?Model monitoring

Common ML design questions

A five-week plan

  1. Week 1: steps 1 and 2. Draw the end-to-end system for a feed ranking model and mark where training-serving skew could creep in.
  2. Week 2: steps 3 and 4. Explain how a model goes from a notebook to a canary deployment, and how you'd roll it back.
  3. Week 3: step 5. Design a recommender for a video app, with a two-tower retriever and a ranker.
  4. Week 4: step 6. For each design so far, name the offline metric, online metric, guardrails and the drift alerts.
  5. Week 5: step 7. Design an internal knowledge assistant with RAG, and estimate its cost per query.

Frequently asked questions

How is ML system design different from regular system design?

Regular system design is about moving and storing data correctly at scale. ML system design adds the question of whether the system's *decisions* are good, which brings in labels, features, training, evaluation and feedback loops. You still need the regular building blocks for the serving and data paths.

Do I need deep learning knowledge?

Enough to choose sensible model families and explain their costs. The interview weighs the system around the model: a simple model with good data, evaluation and monitoring beats a sophisticated one nobody can debug.

Are LLM and RAG questions common now?

Increasingly, especially for AI engineering and applied ML roles. Step 7 covers the patterns they test: retrieval, grounding, evaluation and serving cost.

Further reading