ML Platform Architecture: Self-Service Training, Features and Serving

How companies build an internal ML platform: feature store, training orchestration, model registry, serving, monitoring and the self-service workflow.

Advanced 3 min read · Updated 29 October 2026

Once a company has more than a handful of models, every team rebuilding its own pipelines, feature code and serving stack becomes the bottleneck. An ML platform gives teams shared, self-service building blocks: data and features, training orchestration, experiment tracking, a model registry, serving and monitoring. Uber's Michelangelo, and similar internal platforms at most large tech companies, follow this shape. Interviewers ask about it directly ("design an ML platform") and indirectly whenever you say "the feature store" or "the registry".

The layers of an ML platform

An ML platform's main layers. Data flows left to right into features and training; models flow into a registry and out to serving; monitoring feeds back into retraining.

Each component and what it owns

Component	Owns	Guide
Feature store	Feature definitions, point-in-time training sets, low-latency online lookups	Feature stores
Training service	Job submission, GPU scheduling, distributed training, retries, checkpoints	Distributed training
Pipelines / orchestration	Repeatable DAGs: build data, train, evaluate, register	Retraining
Experiment tracking	Parameters, metrics, artifacts and comparisons for every run	MLOps
Model registry	Versions, lineage, stage (candidate, production), approvals	MLOps
Serving	Online endpoints with autoscaling, batch scoring jobs, shadow and canary routing	Model serving
Evaluation and experiments	Offline evaluation suites and online A/B testing	A/B testing
Monitoring	Latency, errors, feature health, drift, delayed-label quality	Model monitoring

The self-service workflow

What a model team does on a mature platform: declare, train, compare, register and deploy, mostly through configuration rather than custom infrastructure.

Design decisions

Paved road vs flexibility: standard templates for common model types (tabular classifier, ranker, embedding model) cover most teams; an escape hatch lets advanced teams bring custom containers.
Compute scheduling: GPU pools are expensive. Use queues with quotas per team, priorities (production retrains over experiments), preemption with checkpointing, and bin packing on Kubernetes.
Multi-tenancy: isolate teams' data and quotas, and attribute cost per team and per model; see multi-tenancy.
Reproducibility: pin data snapshots, feature versions, code commits and container images for every run.
Governance: approvals for models in regulated areas, model cards, access controls on sensitive features, audit logs.

Scaling numbers to reason with

Organisation : 40 model teams, 300 production models
Retraining   : 200 daily, 100 weekly → ~215 training jobs/day
GPU usage    : avg job 4 GPUs × 3 h = 12 GPU-h → ~2,600 GPU-h/day ≈ 110 GPUs busy
               + experiments (often 2-3× production) → plan ~300-400 GPUs with queuing
Serving      : 300 endpoints; 20 high-QPS (10K+ RPS) need dedicated autoscaled pools,
               the rest share multi-model servers to avoid idle replicas
Registry     : 300 models × ~50 versions kept = 15K versions with lineage metadata

Build vs buy

Managed cloud ML services and open-source stacks (for example Kubeflow, MLflow, Feast, Ray) cover much of this. Companies build their own when scale, latency or integration needs exceed them, but even then they usually assemble open-source pieces. In an interview, naming the layer and its contract matters more than the product.

Where it’s used

Ride-hailing and delivery

One platform serves ETA, pricing, fraud and matching models with shared real-time features.

Social and media

Platforms built around ranking: huge embedding tables, frequent retrains and heavy online experimentation.

Banks and insurers

Platforms that add governance: approvals, explainability reports and audit trails for every model version.

In an interview

Draw the layers: data and features, training, registry, serving, monitoring.
Describe the self-service workflow from config to production.
Address GPU scheduling, quotas and cost attribution.
Explain how reproducibility and lineage are enforced.

Frequently asked questions

What are the components of an ML platform?

A feature store, a training service with GPU scheduling, pipeline orchestration, experiment tracking, a model registry, online and batch serving, evaluation and experimentation, and monitoring.

Why do companies build ML platforms?

So many teams can train, deploy and monitor models through self-service workflows instead of each rebuilding pipelines, feature code and serving stacks, with consistent reproducibility, governance and cost control.