Model Compression: Quantization, Pruning and Knowledge Distillation

Model compression for faster, cheaper inference: post-training quantization and quantization-aware training, pruning, distillation and on-device ML.

Advanced 3 min read · Updated 29 October 2026

Model compression makes a trained model smaller and faster so it fits a latency budget, runs on cheaper hardware or ships to a phone. The three main techniques are quantization (fewer bits per number), pruning (fewer weights) and knowledge distillation (a small model learns from a big one). In system design they are how you answer "this model is too slow or too expensive to serve" without giving up the model.

Why compress

Latency: a ranker scoring 500 candidates in 30 ms, or an LLM generating tokens fast enough to stream.
Cost: serving cost scales with compute and memory; a 4× smaller model can mean a quarter of the GPUs.
Memory: large embedding tables or LLM weights may not fit on one device without it.
On-device ML: keyboards, cameras and offline features need models of a few megabytes that run on a phone's NPU.

Quantization

Store weights (and sometimes activations) in lower precision: from 32-bit floats to 16-bit, 8-bit or 4-bit numbers. Memory and bandwidth shrink proportionally, and integer arithmetic is faster on most accelerators.

Weight memory for a 7-billion-parameter model at different precisions. Halving the bits halves the memory, which is often the difference between needing two GPUs and one.

Approach	How	Trade-off
Post-training quantization (PTQ)	Quantize a trained model, calibrating ranges on a small sample of data	Minutes of work; small accuracy loss at 8 bits, more at 4 bits
Quantization-aware training (QAT)	Simulate low precision during training so the model adapts	Best accuracy at low bits; needs a training run
Weight-only quantization	Quantize weights, keep activations in 16-bit	Common for LLMs, where loading weights is the bottleneck
Mixed precision	Keep sensitive layers (first, last, outlier channels) in higher precision	Recovers accuracy at a small size cost

Pruning

Remove weights that contribute little. Unstructured pruning zeroes individual weights (high sparsity, but needs sparse kernels to see a speed-up). Structured pruning removes whole neurons, attention heads, channels or layers, which shrinks the dense model and speeds it up on ordinary hardware. Prune gradually and fine-tune afterwards to recover accuracy.

Knowledge distillation

Train a small student to imitate a large teacher. The student learns from the teacher's soft probabilities ("70% cat, 25% fox"), which carry more information than hard labels, and often reaches most of the teacher's quality at a fraction of the size. The teacher can also label unlabelled data, giving the student far more training examples.

Distillation: the student is trained on both the true labels and the teacher's soft predictions. Only the student is served.

Other ways to cut serving cost

Cascades: a small model handles easy cases and passes uncertain ones to a big model; see two-stage processing.
Caching predictions or embeddings for repeated inputs.
Compiled runtimes (ONNX Runtime, TensorRT and similar) that fuse operations for the target hardware.
Batching requests on GPUs; see model serving.
Low-rank factorisation of large weight matrices.

On-device vs server-side

	On-device	Server-side
Latency	No network round trip	Network plus queueing
Privacy	Data stays on the phone	Data sent to servers
Model size	Megabytes; heavily compressed	Gigabytes are fine
Updates	Ship with app releases or downloads	Deploy any time
Cost	User's battery and chip	Your servers

Hybrid designs are common: a small on-device model for instant suggestions, a larger server model when the network allows.

Validating a compressed model

Compare offline metrics with the original on the same test set, overall and per slice; compression often hurts rare classes first.
Check calibration; quantized models can shift probabilities.
Measure real latency and memory on the target hardware at production batch sizes.
Shadow-test against the uncompressed model before switching traffic.

Where it’s used

LLM serving

Weight-only 8-bit and 4-bit quantization lets large models fit on fewer GPUs; see LLM serving.

Mobile keyboards and cameras

Distilled, quantized models run on the phone's neural processor for instant, private predictions.

Ranking

A distilled ranker replaces a heavy one inside a tight latency budget, with the heavy model used only offline as teacher.

In an interview

When latency or cost is too high, propose quantization, distillation or a cascade before a weaker model.
Quote the memory arithmetic: parameters × bytes per parameter.
Say how you'd validate accuracy per slice and calibration after compressing.
Weigh on-device against server-side for latency, privacy and update speed.

Frequently asked questions

What is model quantization?

Storing model weights and sometimes activations in lower precision, such as 8-bit or 4-bit integers instead of 32-bit floats, to cut memory and speed up inference with a small accuracy cost.

What is knowledge distillation?

Training a small student model to imitate a larger teacher model's predictions, so the student reaches much of the teacher's quality at a fraction of the size and cost.