Design a Visual Search System: Image Search With Embeddings

ML system design for visual image search like Pinterest Lens: contrastive image embeddings, an ANN index, re-ranking, indexing new images and evaluation.

Advanced 3 min read · Updated 29 October 2026

The question: design a visual search system: a user uploads a photo (or crops part of one) and gets visually similar images or products back, like Pinterest Lens or a shop-the-look feature. The core is an embedding model that maps images to vectors where similar images are close, an approximate nearest neighbour index over billions of images, and a re-ranker that turns similar-looking into useful.

1. Clarify requirements

Question	Answer
Query?	An image or a crop; optionally combined with text ("this but in blue")
Results?	Visually similar images; for shopping, similar purchasable products
Scale?	5B indexed images; 2M new images per day; 5K queries per second
Latency?	Under 500 ms including upload processing
Freshness?	New images searchable within an hour

2. Frame it as an ML problem

Representation learning plus retrieval: learn an encoder f(image) → vector such that similar images have high cosine similarity. Retrieval is nearest-neighbour search; ranking then orders the neighbours by predicted usefulness (for example p(click or save)). See embeddings.

3. Training the embedding model

Contrastive training: two views of the same image (or an image and an image the user engaged with next) are pulled together; other images in the batch are pushed apart.

Self-supervised pairs: augmentations of the same image (crops, flips, colour changes) as positives; no labels needed.
Engagement pairs: a query image and the result the user then saved or clicked, which teaches the product's notion of "similar".
Image-text pairs: training jointly with captions (CLIP-style) enables text and image queries in one space.
Hard negatives: images that look close but were skipped sharpen the model.
Start from a pretrained backbone and fine-tune; training from scratch is rarely needed.

4. Architecture

Indexing (bottom) embeds every new image and inserts it into a sharded ANN index; querying (top) detects objects, embeds the crop, retrieves neighbours, and re-ranks them.

5. Indexing at scale

Vectors     : 5B images × 256 dims
  float32   : 5B × 1 KB ≈ 5 TB      → far too much RAM
  PQ (64 B) : 5B × 64 B ≈ 320 GB    → shard over ~10-20 machines, re-rank top
                                       candidates with full vectors from SSD
Index type  : IVF-PQ (coarse clusters + compressed codes) or HNSW on smaller shards
Query       : fan out to all shards, each returns top 100, merge → top 500
New images  : 2M/day ≈ 23/s → embed on arrival, insert into a small fresh index,
              merged into the main index in nightly rebuilds

Changing the encoder means re-embedding all 5B images into a new index before switching, a large but parallel batch job. See vector databases for index types and the recall-latency trade-off.

6. Re-ranking

Remove near-duplicates (the same image uploaded many times) using a tight similarity threshold or perceptual hashes.
Combine visual similarity with quality and engagement features in a learned ranker.
Apply safety filters and, for shopping, availability and region.
For text-refined queries ("in blue"), combine the image vector with a text vector or filter by attributes.

7. Evaluation

Offline	Online
Recall@k and mAP on human-labelled similar pairs	Click-through and save rate on results
Recall of exact-product matches for shopping	Purchases or outbound clicks from visual search
Per-category slices (fashion, furniture, food)	Query abandonment and reformulation

Where it’s used

Shopping

Find a product from a photo, matched against catalogue images with availability filters.

Inspiration platforms

Related pins and visual discovery from any crop of an image.

Duplicate and copyright detection

The same embedding and index machinery finds near-copies of images.

In an interview

Frame it as representation learning plus nearest-neighbour retrieval plus re-ranking.
Explain contrastive training with augmentation and engagement pairs.
Size the index and use product quantization and sharding.
Plan for re-embedding the corpus when the model changes.

Frequently asked questions

How does visual search work?

An image encoder trained with contrastive learning turns images into embeddings, an approximate nearest neighbour index finds the closest vectors to the query image, and a re-ranker orders the neighbours by usefulness.

How do you search billions of image vectors?

Compress vectors with product quantization, shard the index across machines, query all shards in parallel, and re-rank the top candidates with full-precision vectors.