Design Ad Click Prediction (CTR Prediction): ML System Design

The question: design the system that predicts the probability a user clicks an ad on a social media feed. The prediction feeds the ad auction that decides which ad is shown, so it must be accurate, calibrated, fresh, and fast enough to score thousands of ads per request. Ads ranking is one of the most commonly asked ML design questions at large ad-funded companies.

1. Clarify requirements

2. Frame it as an ML problem

Binary classification: given (user, ad, context), predict p(click). Because the score multiplies a bid, the probability itself must be right, not just the order. If pCTR is 2× too high for one advertiser, they win auctions they shouldn't and pay for it. So calibration is a first-class requirement, alongside log loss and AUC.

expected value (eCPM) = bid_per_click × pCTR × 1000
ad A: bid $2.00, pCTR 1.5% → eCPM $30
ad B: bid $0.50, pCTR 8.0% → eCPM $40   ← wins despite a lower bid
(a production auction also adds quality and user-experience terms)

3. Data

• Rows: one per impression, labelled 1 if clicked within a window (for example 15 minutes), else 0.
• Imbalance: CTR around 1%, so downsample negatives (keep 10%) and recalibrate afterwards; see training data.
• Volume: 100B impressions/day; even downsampled, billions of rows per day.
• Delayed conversions: for conversion models, either wait for a window (stale) or train on what's known and correct with a delay model.
• Click fraud and bots: filter invalid clicks before training, or the model learns to target bots.

4. Features

Ads models are dominated by sparse id features with billions of values. Hashing and embedding tables handle them; see feature engineering. Smoothed CTR counters use priors so a new ad with 1 click in 2 impressions doesn't look like a 50% CTR.

5. Model

1. Baseline: logistic regression on hashed features and crosses. Calibrated, fast, trains online. A classic design used gradient-boosted trees to create features for a logistic regression.
2. Production: a deep model with embedding tables for sparse features, an interaction layer (feature crosses, dot products) and an MLP, in the wide-and-deep or DLRM family.
3. Multi-task: heads for click and conversion, sharing embeddings.
4. Calibration layer: isotonic regression or Platt scaling on recent data, per segment if needed.

6. Serving architecture

Requests   : 100B impressions/day ≈ 1.2M ad slots/s avg (peaks ~3M/s)
Scoring    : 500 ads × 1.2M = 600M predictions/s → GPU batching across requests
Embeddings : 2B hashed ids × 64 dims × 4 B ≈ 512 GB → sharded parameter servers,
             hot ids cached on the inference hosts
Budget     : targeting 15 ms | features 15 ms | scoring 30 ms | auction 5 ms

7. Training and freshness

New ads and campaigns appear constantly, and click patterns shift by the hour, so ads models are among the most frequently updated in industry. A typical setup: incremental training every few hours (or continuous online learning) from the click stream, plus a periodic full retrain. Each update is validated on a recent holdout before it's pushed. See continual learning.

8. Evaluation

• Offline: normalised log loss (log loss relative to predicting the average CTR), AUC, and calibration (predicted vs observed CTR in buckets; ratio close to 1.0 overall and per advertiser). See ML evaluation.
• Online: revenue per thousand impressions, CTR, advertiser conversions and return on ad spend, plus user guardrails: ad hide rate, session length.
• Experiments: user-level A/B tests, and budget-split tests so advertisers' budgets don't leak between arms.

9. Monitoring

• Calibration ratio per advertiser and placement, alerted when it drifts beyond ±5%.
• Feature freshness of real-time counters; a stale counter stream silently hurts revenue.
• Sudden CTR spikes on an advertiser (click fraud) and invalid-traffic rates.
• Score distribution and revenue per request compared to the same hour last week.

Follow-up questions to expect

• *Why does calibration matter if AUC is high?* AUC measures order only; the auction multiplies probabilities by bids, so miscalibration misprices ads.
• *How do you handle a new ad?* Priors from advertiser and category, content embeddings of the creative, and exploration traffic.
• *How do you predict conversions that arrive days later?* A separate conversion model with delayed-feedback handling, trained on post-click data.