A/B Testing and Experimentation Platforms

An A/B test (online controlled experiment) randomly splits users between the current experience (control) and a change (treatment) and compares outcomes, so the difference can be attributed to the change rather than to seasonality or luck. An experimentation platform makes that safe and routine at scale: consistent assignment, exposure logging, metric computation, statistics and guardrails, for hundreds of concurrent experiments.

The pieces of an experiment

• Hypothesis and primary metric: "The new ranker increases purchases per user." Decide this before launch.
• Randomisation unit: usually the user (logged-in ID or a device ID); sometimes the session, the request, or a cluster such as a city.
• Allocation: for example 50/50, or 5% treatment while ramping.
• Guardrail metrics: latency, errors, revenue and complaints that must not get worse.
• Duration and sample size: fixed in advance from the minimum effect worth detecting.

Assignment that stays consistent

Assign by hashing the unit ID with an experiment-specific salt: bucket = hash(experiment_id + user_id) mod 1000. The same user always lands in the same bucket, without storing assignments, and different experiments get independent splits because their salts differ.

Log an exposure event only when the user actually encounters the changed experience, and analyse those users. Analysing everyone assigned, including people who never reached the feature, dilutes the effect.

Statistics that matter in practice

• Sample size and power: smaller effects need many more users. Detecting a 1% lift needs roughly four times as many users as detecting a 2% lift. Compute the sample size before launch.
• Don't peek and stop early: checking p-values daily and stopping at the first significant result inflates false positives. Use a fixed horizon, or sequential testing methods designed for continuous monitoring.
• Multiple comparisons: testing twenty metrics means one will look significant by chance. Pre-register the primary metric.
• Variance reduction: methods such as CUPED use each user's pre-experiment behaviour to reduce noise, so experiments finish sooner.
• Sample ratio mismatch (SRM): if a 50/50 split shows 50.8/49.2 with millions of users, assignment or logging is broken. Check it automatically and distrust results that fail.

When user-level A/B tests break

Running many experiments at once

• Layers: experiments on unrelated surfaces run in independent layers, each with its own hash salt, so a user can be in many at once.
• Mutual exclusion: experiments that touch the same feature share a layer and get disjoint traffic.
• Config, not deploys: allocation lives in dynamic config (see feature flags), so ramping or killing an experiment doesn't need a release.
• Automated guardrails: the platform can stop an experiment that breaches a guardrail threshold.