Continual Learning and Model Retraining: When and How to Retrain

Model retraining strategies: scheduled vs triggered retraining, stateless vs incremental training, online learning, training pipelines and safe promotion.

Intermediate 3 min read · Updated 29 October 2026

A model starts going stale the day it ships: users change, items change, fraudsters adapt. Continual learning is the practice of retraining and redeploying models regularly, from a weekly batch retrain up to models updated every few minutes. In design interviews you should state how often the model retrains, what triggers it, how it trains (from scratch or incrementally), and how a new version is promoted safely.

Why models need retraining

Data drift: the input distribution changes (new markets, a new app version, seasonality).
Concept drift: the relationship between inputs and labels changes (what counts as fraud this month).
New entities: new videos, products and users the old model has never seen, the cold start problem.
Freshness pays: in fast domains (news, ads, short video) a model trained on yesterday's data measurably beats one trained last week.

Detecting drift is covered in model monitoring; this page is about the response.

Retraining strategies

Strategy	How	Fits	Cost
Manual / ad hoc	Someone reruns the notebook	Prototypes only	Low effort, high risk
Scheduled batch	Retrain from scratch daily or weekly on a sliding window	Most classifiers and rankers	Moderate, predictable
Triggered	Retrain when drift or quality metrics cross a threshold	Stable domains with occasional shifts	Low when calm; needs good monitoring
Incremental (warm start)	Continue training the last model on only the new data	Large models where full retrains are expensive	Cheap per update; risk of slowly accumulating errors
Online learning	Update weights on a stream, minutes after events	Ads CTR, news, trending	Complex infrastructure and safeguards

Stateless retraining (top lane) rebuilds the model from a sliding window each time; stateful retraining (bottom lane) warm-starts from the previous model and only adds the newest data.

Many teams combine them: incremental updates daily, plus a full retrain from scratch weekly to reset any accumulated drift and pick up feature or architecture changes.

The automated training pipeline

A continual training pipeline. Every step is automated; a candidate model is only promoted if it passes offline checks, then a shadow or canary stage, against the current production model.

Orchestration: the pipeline is a DAG in an orchestrator such as Airflow, Kubeflow Pipelines or similar; see pipeline orchestration.
Champion vs challenger: the new model must beat the current one on the same, freshest held-out data, overall and on key slices.
Validation gates: schema checks on data, minimum sample counts, metric thresholds, calibration checks, and prediction distribution sanity.
Registry and lineage: each version is stored with its data snapshot, code commit and metrics; see MLOps.
Safe rollout: shadow traffic, then a canary percentage, then full traffic; automatic rollback on guardrail breaches.

Online learning in practice

Ads and news models sometimes update weights continuously from a click stream. The system looks like a stream job that joins impressions to clicks within a label window, produces mini-batches, and updates a parameter server; the serving fleet pulls fresh weights every few minutes.

Delayed labels: you can't wait forever for a click. Train on impressions after a short window and correct later, or model the delay explicitly.
Poisoning and bugs: a broken logger or a bot attack can corrupt a model in minutes. Keep snapshots, validate each update on a holdout stream, and alert on sudden metric moves.
Reproducibility: the model never exists as one clean training run, so snapshot weights regularly.

Choosing a cadence

Measure it rather than guess: train models on data ending at different dates and evaluate them all on the same recent week. If a model trained one day ago beats one trained seven days ago by a meaningful margin, retrain daily. If the curve is flat, weekly or monthly is enough, and cheaper.

Freshness experiment (offline NDCG@10 on the latest week)
  trained through yesterday   0.412
  trained through 3 days ago  0.409
  trained through 7 days ago  0.401
  trained through 30 days ago 0.383
→ the first week loses ~2.7%; daily retraining is worth its cost here

Where it’s used

Ads click prediction

Frequent incremental updates because new ads and campaigns appear constantly and click patterns shift by the hour.

Fraud detection

Scheduled retrains on labelled data weeks old, with rules updated faster for newly seen attacks.

Search ranking

Weekly or monthly retrains; query intent drifts slowly, apart from news and seasonal queries.

In an interview

State a retraining cadence and justify it with a freshness experiment.
Distinguish full retrains from incremental updates, and say when you'd combine them.
Describe champion-vs-challenger evaluation and the gates before promotion.
Cover rollback and what happens if a bad batch of data reaches training.

Frequently asked questions

How often should a machine learning model be retrained?

Measure it: train models on data ending at different dates and evaluate them on the same recent period. If freshness improves metrics meaningfully, retrain often; if not, weekly or monthly is enough.

What is the difference between stateless and stateful retraining?

Stateless retraining rebuilds the model from scratch on a sliding window of data; stateful (incremental) retraining continues training the previous model on only new data, which is cheaper but can accumulate errors.