Embeddings in ML Systems: How to Train, Serve and Version Vectors

An embedding is a dense vector of numbers (typically 64 to 1,536 of them) that represents an item, a user, a query, a piece of text or an image, so that similar things end up close together. Embeddings are how ML systems handle ids with millions of values, compare text and images by meaning, and retrieve a thousand good candidates out of a billion in milliseconds. This guide covers how they are learned, stored, served and refreshed, which is what system design interviews probe.

What an embedding gives you

• Compression: a user id with 500 million values becomes a 128-number vector instead of a 500-million-wide one-hot column.
• Similarity: distance between vectors (cosine similarity or dot product) measures relatedness, so "find items like this" becomes a nearest-neighbour search.
• Transfer: an embedding learned on one task (say watch history) is a strong feature for another (say notifications).
• Cross-modal matching: text and images embedded in one space let a text query find photos.

How embeddings are learned

The two-tower model

The workhorse of retrieval. One tower encodes the query side (user, context, or search text), the other the item side. Training uses (user, item engaged) pairs as positives and other items in the batch as negatives. Because the towers only meet at a final dot product, item vectors can be precomputed and indexed, and only the user vector is computed at request time.

In-batch negatives over-penalise popular items (they appear as negatives more often), so production systems correct the logits by item frequency, and mix in hard negatives: items that look relevant but weren't chosen.

Serving embeddings

• Item vectors live in an approximate nearest neighbour index (HNSW, IVF-PQ) for retrieval, and in a key-value store for lookup as ranking features.
• User vectors are either precomputed daily (cheap, stale) or computed per request from recent behaviour (fresh, costs a model call).
• Embedding tables inside ranking models can be huge: 1 billion ids × 64 floats × 4 bytes is 256 GB, so they are sharded across parameter servers or GPUs, and rare ids are hashed together.
• Quantisation (float32 to int8, or product quantisation) cuts index memory 4× or more with a small recall loss.

Index sizing: 200M items × 256 dims
  float32 : 200M × 256 × 4 B ≈ 205 GB   → shard across ~4-8 machines
  int8    : 200M × 256 × 1 B ≈  51 GB   → fits on 1-2 large machines
  PQ (32 B per vector)        ≈ 6.4 GB  → one machine, lower recall; rerank exact
Query cost: ~1-10 ms per ANN search at recall@100 ≈ 0.95 (tuned)

Refreshing and versioning

Embeddings from different model versions live in different spaces: a user vector from model v2 compared to item vectors from v1 is meaningless. So:

1. Tag every stored vector with the model version that produced it.
2. On a new model, re-embed the whole corpus into a new index in the background.
3. Switch user-side and item-side together (blue-green), and keep the old index until the new one is verified.
4. New items between full rebuilds are embedded on arrival with the current model and inserted incrementally.

Re-embedding a large corpus is a real batch job: 200M items at 2,000 items per second per GPU is about 28 GPU-hours. Budget for it whenever you change the encoder.

Evaluating embeddings

• Retrieval metrics: recall@k against held-out engagements (did the item the user actually watched appear in the top 500?).
• Downstream impact: offline ranking metrics or an online test using the new candidates.
• Sanity checks: nearest neighbours of known items, clustering by category, and drift in the vector distribution between versions.