Data Labeling for ML: Human Labels, Weak Supervision and Active Learning

How ML systems get labels: natural labels from behaviour, human annotation, label quality, weak supervision, active learning and LLM-assisted labeling.

Intermediate 3 min read · Updated 29 October 2026

Every supervised model needs labels: the ground truth it learns to predict. Data labeling is how a system gets them, either for free from user behaviour (natural labels) or by paying people and programs to produce them (hand labels, weak supervision, active learning, and increasingly LLM-assisted labeling). In an ML system design interview, "where do the labels come from, how good are they, and how fast do they arrive?" is a question you should answer before you're asked.

Natural labels vs hand labels

	Natural labels	Hand labels
Source	User behaviour: clicks, purchases, skips, chargebacks	Annotators, domain experts, crowd workers
Cost	Almost free	Expensive per label; experts much more so
Volume	Huge	Limited by budget and throughput
Speed	Seconds (click) to months (chargeback)	Hours to weeks per batch
Noise	Biased by what was shown; implicit and ambiguous	Annotator disagreement, guideline drift
Typical use	Ranking, ads, recommendations, ETA	Content moderation, medical, search relevance, NER

Natural labels carry the system's own bias: you only observe outcomes for items you showed. That's a feedback loop you have to correct for. They also have a feedback delay, which is why fraud models train on data that's weeks old.

A human labeling pipeline

A production labeling pipeline. Items are sampled smartly, labelled by more than one person when it matters, checked against gold questions, and only agreed labels enter training.

Guidelines: a written policy with examples and edge cases. Most label noise is guideline ambiguity, not carelessness.
Redundancy: two or three labels per item for hard tasks; measure inter-annotator agreement (for example Cohen's kappa) and send disagreements to an expert.
Gold questions: items with known answers mixed into the queue to score each annotator and catch drift.
Tooling: a task queue with leasing (so two people don't take the same item), autosave, keyboard shortcuts and audit logs, the same queue patterns as any work system.
Privacy: redact personal data before items reach annotators, and restrict access to sensitive queues.

Getting more labels for less

Active learning

Instead of labelling random items, label the ones the current model is least sure about (scores near the decision boundary), or where several models disagree. Each label then teaches the model more. Keep a random slice too, so evaluation data stays unbiased.

Weak supervision

Write many cheap, noisy labeling functions (keyword rules, heuristics, existing models, knowledge bases), then combine their votes with a label model that learns how much to trust each. You get large, noisy training sets quickly; a small hand-labelled set still measures quality.

LLM-assisted labeling

A large language model can pre-label text, images or conversations with a rubric, and humans review instead of starting from scratch. Calibrate it first: compare LLM labels to expert labels on a few hundred items per class, keep humans on classes where agreement is low, and re-check whenever the prompt or model changes. The same judging ideas appear in LLM evaluation.

Semi-supervised and transfer learning

Pretrained models and embeddings need far fewer labels to fine-tune. Pseudo-labelling (training on a model's confident predictions) can stretch a small labelled set further, with the risk of reinforcing its mistakes.

Label quality and drift

Measure noise: re-label a sample periodically; track agreement per class and per annotator group.
Version labels with the policy: when the definition of "harmful" changes, old labels may no longer be correct.
Slice audits: check that labels for minority languages or regions are as accurate as the rest.
Feedback from production: appeals, reversed decisions and user reports are high-value labels for retraining.

Estimating labeling cost

Goal            : 200K labelled posts for a new harm classifier
Labels per item : 3 (agreement needed)        → 600K judgements
Time per item   : 20 s                        → 600K × 20 s ≈ 3,333 hours
Annotators      : 50 people × 6 productive h/day → 300 h/day
Duration        : 3,333 ÷ 300 ≈ 11 working days
Cut it          : active learning picks 60K items that matter most
                  + LLM pre-labels shorten review to ~8 s/item
                → 60K × 3 × 8 s ≈ 400 hours ≈ 1.5 days

Numbers like these let you argue for active learning or pre-labelling with something more convincing than "it's cheaper".

Where it’s used

Content moderation

Policy experts write guidelines, vendors label at scale, appeals feed back as corrections, and active learning targets new abuse patterns.

Search relevance

Raters grade query-document pairs on a relevance scale; click logs supplement them for scale.

Self-driving and vision

Model pre-labels bounding boxes and humans correct them, with hard examples prioritised by uncertainty.

In an interview

Say whether labels are natural or need annotation, and how long they take to arrive.
For hand labels, mention guidelines, redundancy, gold questions and agreement.
Propose active learning or pre-labelling to cut cost, with a random slice for unbiased evaluation.
Explain how labels are versioned when the policy changes.

Frequently asked questions

What is the difference between natural labels and hand labels?

Natural labels come from user behaviour, such as clicks, purchases or chargebacks, and are cheap and plentiful but biased and sometimes delayed. Hand labels come from annotators or experts, are more precise, and are slow and expensive.

What is active learning?

A labelling strategy that sends the examples the current model is least sure about to annotators first, so each label improves the model more. A random sample is still kept for unbiased evaluation.