ML Model Serving: Batch vs Real-Time Inference

ML model serving explained: batch vs online inference, latency budgets, dynamic batching, GPU utilisation, model versioning, shadow and canary deployments.

Intermediate 3 min read · Updated 24 October 2026

Model serving is how a trained model produces predictions for real users and systems. Predictions can be made in batch, precomputed for many entities on a schedule, or online, computed per request within a latency budget. Online serving adds the problems of any low-latency service (scaling, tail latency, failures) plus ML-specific ones: fetching features, using accelerators efficiently, and rolling out new model versions safely.

Batch vs online inference

	Batch inference	Online (real-time) inference
When predictions are made	On a schedule, for all or many entities	Per request, when the user or system asks
Latency	Minutes to hours; nobody waits	Milliseconds to a second
Freshness	As old as the last run	Uses the latest request and context
Cost profile	Efficient: large batches on cheap capacity	Must be provisioned for peak traffic
Failure impact	A delayed run; serve the previous results	Requests fail or fall back immediately
Examples	Nightly churn scores, weekly product recommendations by email	Fraud checks at payment time, search ranking, chat responses

Many systems mix both: precompute what you can (embeddings, candidate lists, scores for inactive users), and compute online only what depends on the moment. A third option is streaming inference, where a stream processor scores events as they arrive and writes results to a store.

An online serving path

A prediction request: the caller hits a model service, which fetches features, batches inference on accelerators and logs inputs and outputs for monitoring.

Latency and throughput

Budget the whole path: network, feature lookup, preprocessing, inference and postprocessing. A 100 ms budget might leave 40 ms for the model itself.
Dynamic batching: accelerators are efficient on batches, but requests arrive one at a time. Inference servers queue requests for a few milliseconds and run them together, trading a little latency for much higher throughput.
Model optimisation: quantisation (lower-precision weights), distillation into a smaller model, compiled runtimes and pruning cut latency and cost.
Tail latency: a slow feature store or garbage-collection pause shows up at p99. Set timeouts and a fallback for every dependency; see timeouts and retries.
Right hardware: small tree models are cheapest on CPUs; large neural networks need GPUs or other accelerators, which are expensive to leave idle.

Deploying new versions safely

Strategy	How it works	What it catches
Shadow (dark launch)	New model receives a copy of live traffic; its predictions are logged, not used	Crashes, latency and prediction distribution differences, with no user impact
Canary	A small percentage of traffic uses the new model, rising if metrics hold	Real-world regressions at limited blast radius
A/B test	Users are randomly split between models and business metrics are compared	Whether the new model actually improves outcomes
Blue-green	Switch all traffic to a fully provisioned new version, keeping the old one for rollback	Fast cutover and rollback

Version every model artifact in a model registry, keep the previous version warm for instant rollback, and record which model version made each prediction. See also deployment strategies and A/B testing.

Failure handling

Fallbacks: if the model times out, return a simpler model's answer, a cached prediction, or a safe default (allow the payment and flag for review, show popular items).
Input validation: reject or default malformed inputs rather than producing confident nonsense.
Autoscaling: scale on queue depth or GPU utilisation, and keep warm capacity because model loading can take minutes. See autoscaling.
Monitoring: watch latency, error rates and prediction distributions, and feed logged predictions into model monitoring.

Where it’s used

Payments fraud

Online models score each transaction within tens of milliseconds, falling back to rules if the model is unavailable.

Marketing

Batch jobs score every customer nightly for churn risk and write results to the CRM.

Search and feeds

Ranking models score hundreds of candidates per request with dynamic batching on GPUs.

In an interview

Choose batch or online from freshness and latency needs, and precompute where possible.
Break down the latency budget and mention dynamic batching.
Roll out with shadow traffic and canaries, with instant rollback.
Define a fallback for timeouts and log every prediction with its model version.