Machine Learning System Architecture: Data to Production Models
How production ML systems connect data pipelines, training, feature stores, model registries, deployment, online serving, and drift monitoring.
A production machine-learning system includes much more than a model. It moves data into training and feature pipelines, records model versions and evaluation results, deploys an artifact safely, serves predictions within a latency and cost budget, and observes whether the model still works on live traffic.
The training-to-serving lifecycle
The most important boundary is offline versus online. Offline jobs can process large historical datasets and tolerate minutes or hours. Online inference must meet the product's request latency and availability targets. The two paths must agree on feature definitions and transformations, or training-serving skew can make a well-scoring model fail in production.
Keep data, features and models reproducible
- Version training code, input snapshots, feature definitions, parameters, and model artifacts; record them together with evaluation results.
- A feature store can share definitions between offline training and online inference, but it adds a serving system and consistency contract. Introduce it when reuse or skew is a real problem.
- For a label, define when it becomes available. Training on information that arrives after the prediction time creates leakage.
- Make training jobs restartable and backfillable. Store intermediate datasets or manifests when recomputing the full pipeline is expensive.
- A model registry should track approval state, compatibility, provenance, and rollback-ready prior versions, not just a file location.
| Workload | Serving shape | Trade-off |
|---|---|---|
| Low-latency online prediction | Replicated model service with autoscaling and feature lookup | Tight latency; control warm-up, concurrency, and feature-store dependency |
| Large offline scoring job | Batch workers reading a snapshot and writing results | High throughput; easier retries and cost-efficient accelerators |
| Small model or rules | In-process inference in the application | Fewer network hops; model release is coupled to application release |
| Expensive or variable inference | Queue with bounded workers and status polling | Protects capacity; adds asynchronous product behavior and queue delay |
Release safely and meet product behavior
Expose a stable prediction API that includes a model version, confidence or explanation metadata when useful, and a clearly defined timeout. Deploy a candidate behind a shadow or canary path, compare its quality and latency with the current version, then increase traffic gradually. Keep a fast rollback and a non-ML fallback for cases where the model or feature service is unavailable.
- Capacity: measure requests per second, model load time, memory, accelerator utilization, and the cost of idle replicas.
- Latency: budget separately for authentication, feature lookup, preprocessing, inference, and post-processing.
- Reliability: set a timeout, bound retries, and avoid retry storms on GPU or feature-store overload.
- Privacy: minimize training and inference data, enforce access controls, and define retention for prompts, features, labels, and predictions.
- Human review: for consequential predictions, define thresholds, appeal paths, and who can override the output.
Monitor the model and the pipeline
Infrastructure health and model quality are different signals. Track request success and latency, but also input distributions, missing-feature rates, prediction distributions, and delayed ground-truth metrics. Drift is a reason to investigate, not proof that quality has fallen; compare against labels and segment results by user cohort or operating condition. Automate retraining only when the data, evaluation, and approval gates are trustworthy.
A model that is statistically accurate can still be a bad product. Define the business outcome, failure cost, fairness constraints, and fallback behavior before optimizing a single offline metric.
A practical interview sequence
- Clarify the prediction task, label timing, online or batch latency, quality metric, traffic, and failure cost.
- Draw the path from source events through feature generation and training to a versioned model.
- Explain how online features match training features and how model releases are evaluated and rolled back.
- Add timeouts, fallbacks, privacy boundaries, and capacity controls to inference.
- Close with drift and outcome monitoring, retraining criteria, and human review where the decision is high impact.
Where it’s used
A batch training pipeline can refresh candidate models while an online service combines the active model with fresh user and item features.
A low-latency model can score a transaction using recent signals, with a rules-based or manual-review path when confidence is low.
Candidate retrieval and model ranking can scale independently, with model versions and relevance outcomes monitored by cohort.
In an interview
- Draw data preparation, training, model registry, deployment, inference, and feedback as separate lifecycle steps.
- Call out training-serving skew and how shared feature definitions or parity tests detect it.
- Give the inference path a latency budget, timeout, capacity limit, and fallback.
- Separate infrastructure monitoring from delayed model-quality and drift signals.