Design Harmful Content Detection: ML Content Moderation System

The question: design a system that detects harmful content (hate speech, violence, nudity, spam, scams, self-harm) on a social platform where users upload text, images and video. It must catch most violations before many people see them, avoid removing legitimate posts, route hard cases to human reviewers, and keep up with adversaries who change tactics. It's a common integrity question at social media companies.

1. Clarify requirements

2. Frame it as an ML problem

Multilabel classification: one post can violate several policies. Output a probability per harm category. Each category gets its own thresholds, because the cost of errors differs: a missed child-safety violation is far worse than a missed spam post, while over-removing political speech has its own serious cost.

3. Architecture

4. Models

• Hash matching: perceptual hashes of known violating images and videos match re-uploads even after resizing or cropping. Cheap, precise, first line.
• Unimodal encoders: a multilingual text encoder, an image encoder, and a video model on sampled frames plus audio transcripts.
• Fusion: a classifier over concatenated embeddings (and metadata) so a harmless image with a hateful caption is caught.
• Multi-task heads: one per harm category sharing the encoders; adding a new policy means adding a head and labels, not a new system.
• Account and behaviour signals: new account, posting rate, prior violations, network of known bad actors; often more robust than content alone against spam.
• Foundation models: LLMs or vision-language models with a policy prompt can label new harm types quickly and serve as teachers for smaller, cheaper production classifiers; see model compression.

5. Labels

Human labels are central here. Policy teams write guidelines; trained reviewers label; disagreements are adjudicated. Sampling for labelling uses active learning around the thresholds and over-samples rare harms and languages. See data labeling.

Policies change. When the definition of a harm changes, old labels can become wrong. Version labels with the policy version and retrain from relabelled data.

6. Serving and scale

Uploads        : 1B/day ≈ 12K/s avg, ~30K/s peak
Text           : cheap; every post and comment scored synchronously at upload
Images         : ~400M/day; image encoder on GPU, batched, ~50 ms
Video          : ~50M/day; sample frames + transcribe audio asynchronously;
                 distribution held back or limited until scored
Re-scoring     : when a post gets reports or goes viral (e.g. 10K views/hour),
                 re-score with a heavier model and prioritise review
Review queue   : ~1-2% of uploads flagged → 10-20M/day; only the top ~2M by
                 (severity × p × predicted reach) reach humans; the rest are actioned
                 by thresholds or reduced in distribution

• Prioritise review by expected harm: severity × probability × predicted views. A borderline post from an account with millions of followers goes first.
• Reduce distribution while uncertain, instead of choosing only between removal and full reach.
• Reviewer wellbeing: blur graphic media by default, cap exposure per reviewer, rotate queues.

7. Evaluation

8. Adversaries and monitoring

• Bad actors misspell words, add noise to images and use coded language. Augment training data with these perturbations and refresh labels from recent reports.
• Monitor per-harm score distributions, report rates and prevalence weekly; a jump in reports with flat model flags means a new pattern.
• Rapid response: analysts can add hashes, keywords or rules in minutes while models are retrained.