Training Data for ML Systems: Sampling, Splits, Imbalance and Leakage
Building ML training data: data sources, sampling strategies, time-based splits, class imbalance, negative sampling, data leakage and dataset versioning.
In production ML, the training data usually matters more than the model. This guide covers how training examples are built from logs, how to sample them, how to split them without peeking into the future, how to handle class imbalance, and the forms of data leakage that make a model look excellent offline and fail on day one.
Where training examples come from
Most ML systems build examples by joining what the system showed or decided with what happened next. A recommender logs every impression, then joins clicks and purchases onto it. A fraud model logs every scored transaction, then joins chargebacks weeks later.
- Label window: how long to wait for an outcome. A click within 30 minutes, a purchase within 7 days, a chargeback within 90 days. Too short loses positives; too long delays training.
- Point-in-time correctness: features must be read as of the impression time, not today. A feature store does this join for you.
- Logging the decision context: position on screen, model version, and the score the model gave. You need them later for debiasing and debugging.
Sampling strategies
| Strategy | How it works | When to use |
|---|---|---|
| Random | Every example has the same chance | Default when data is plentiful and balanced |
| Stratified | Sample within each class or segment | Keep rare classes and small countries represented |
| Negative downsampling | Keep all positives, a fraction of negatives | Click and fraud data with very few positives |
| Reservoir | Keep a fixed-size uniform sample of a stream | Sampling from an unbounded event stream |
| Importance | Weight examples by how likely they were to be sampled | Correcting for a biased logging policy |
| Hard negative mining | Pick negatives the model currently confuses | Training retrievers and embedding models |
Downsampling negatives changes the base rate the model sees, so its raw probabilities are too high. If anything downstream needs real probabilities (ad auctions, thresholds), recalibrate: with negatives kept at rate w, the corrected probability is p / (p + (1 − p) / w).
Splitting: train, validation, test
Random splits leak information whenever the data has time or identity structure. In production systems the default is a time-based split: train on the past, validate and test on what came after.
- Group splits: if one user contributes many rows, keep all of a user's rows in the same split, or the model memorises the user.
- Rolling (backtest) evaluation: retrain and test across several windows to see how stable the gain is.
- Slices: report metrics for new users, small markets, and each device type, not only the average.
Class imbalance
Fraud might be 0.1% of payments and ad clicks 1% of impressions. A model that always predicts the majority class looks accurate and is useless. Options, roughly in the order to try them:
- Use the right metric: precision, recall and PR-AUC instead of accuracy. See ML evaluation.
- Choose the threshold from the cost of each error, not 0.5.
- Reweight the loss so minority errors cost more (class weights, focal loss).
- Resample: downsample the majority or oversample the minority (synthetic oversampling is rarely needed with large data).
- Collect more positives: route uncertain cases to reviewers, which also feeds labelling.
Data leakage
Leakage is when a feature contains information that won't be available at prediction time, or that encodes the label. Offline metrics become unbelievably good; production metrics collapse.
| Leak | Example | Fix |
|---|---|---|
| Future data in features | "Orders in the last 7 days" computed at training time includes orders after the impression | Point-in-time feature joins |
| Label proxy as feature | A refund_issued flag used to predict fraud | Audit features against the label's causal path |
| Random split on time data | Training on Friday, testing on Thursday | Time-based splits |
| Duplicates across splits | The same image or user in train and test | Deduplicate; split by group |
| Preprocessing on all data | Normalising with the mean of the full dataset | Fit transforms on train only |
Dataset versioning and lineage
Every trained model should point to the exact dataset snapshot, feature definitions and code it came from, so you can reproduce it, compare it and roll back. Store datasets as immutable, versioned tables (for example with an open table format) and record the version in the model registry.
Where it’s used
Impression logs joined to clicks and conversions, with heavy negative downsampling and calibration afterwards.
Labels arrive weeks later, so training uses data old enough to be labelled and features rebuilt point-in-time.
Strictly time-based splits and rolling backtests, since the future always looks different from the past.
In an interview
- Describe exactly how a training row is built: the unit, the label window and the feature timestamp.
- Use a time-based split and explain why a random split would leak.
- If you downsample negatives, say how you recalibrate.
- Name one leakage risk specific to the problem and how you'd prevent it.
Frequently asked questions
What is data leakage in machine learning?
Data leakage is when a training feature contains information that will not be available at prediction time or that encodes the label, such as aggregates computed with future events. It makes offline metrics look excellent while the model fails in production.
Why use a time-based split instead of a random split?
Because production models predict the future from the past. A random split lets the model train on events that happened after the ones it is tested on, which inflates offline metrics.
How do you handle class imbalance?
Use metrics such as precision, recall and PR-AUC instead of accuracy, choose thresholds from the cost of each error, reweight the loss, downsample the majority class (and recalibrate probabilities), and collect more positive examples.
Further reading
- Designing Machine Learning SystemsChip Huyen, O'Reilly
- Practical Lessons from Predicting Clicks on Ads at FacebookHe et al., ADKDD 2014