Caching: How It Works, Where It Lives and Why It Matters

A cache keeps a copy of the result of expensive work (a database query, a computation, a remote call, a file) somewhere faster, so repeated requests skip the work. A user opens a product page and the database query takes 50 ms. A second later another user opens the same product, and the database runs the identical query for the identical row. Thousands of users view that product every minute. Serving it from memory instead is orders of magnitude faster. Caching is usually the single biggest lever for read-heavy systems; its price is that cached data can be stale.

The speed gap

Caching starts with how differently storage performs. Each step down the hierarchy adds a big jump in latency, sometimes 10×, sometimes orders of magnitude (1 ms = 1,000,000 ns):

RAM versus a disk seek is a gap of five orders of magnitude. Serving from memory instead of disk can multiply the throughput of the same hardware, and often decides whether a system survives production load or collapses under it. See latency numbers.

Why caching works: power laws

If every request were for something different, caching would be useless. In practice, most read-heavy workloads follow power-law (Zipf) distributions: a small fraction of the data serves most of the requests.

• A video platform hosts 10 million videos; the top 1,000 get 60% of all views.
• An e-commerce site lists millions of products; the homepage and top categories produce 80% of page loads.
• A social network stores billions of posts; trending content drives most reads.

So a small cache has an outsized effect: in a highly skewed catalog, caching just the top 1,000 of 10 million items can absorb most reads. How much depends on how steep the popularity curve is, but real-world popularity is uneven, never uniform.

The mental model: time-bounded copies

A cache is a time-bounded copy of data stored closer to the requester. Each part of that definition matters:

• Copy. In typical designs the cache is not the source of truth; the database is. The cache holds a duplicate that can drift from the original after an update.
• Time-bounded. Entries don't live forever. Most expire after a set time (a TTL, time-to-live); others are explicitly invalidated when the source changes. Either way the cache should eventually reflect reality, though a failed invalidation can leave stale entries behind. Without any expiry, stale data would persist forever.
• Closer to the requester. Fewer network hops, faster storage, or both. A browser cache is on the user's device (no network at all). A CDN is in a nearby data center (around 10 ms instead of 150 ms across a continent). An application cache is in RAM: microseconds in-process, sub-millisecond to a few milliseconds for a networked cache like Redis, all far faster than a disk-bound database query.

In one sentence: pay a small freshness cost to avoid fetching the same data from a slow source again and again.

Three benefits

• Lower latency. A hit returns from memory in microseconds or a millisecond; a miss costs a database round trip in milliseconds. Across a page that makes 20 backend calls, saving 40 ms per call saves nearly a second.
• Higher throughput. The database sees far fewer requests. If 90% of reads hit the cache, a database rated for 5,000 queries a second effectively supports 50,000. The same hardware serves 10× the users without new replicas or bigger instances.
• Lower cost. Fewer queries means less compute and I/O. A system that would need 10 read replicas at full load might need 2. Memory costs more per GB than disk, but a few GB of cache replacing terabytes of repeated disk reads is a net saving.

In interviews, frame a caching decision around which benefit matters most: a real-time game wants latency, a high-traffic API wants throughput, a cost-sensitive startup wants savings.

How hit rate changes everything

Average latency = hit rate × cache latency + miss rate × database latency, and database load = total load × miss rate. With an (illustrative) 0.5 ms networked cache, a 50 ms database and 10,000 requests a second:

def simulate(hit_rate_percent, cache_ms=0.5, db_ms=50.0, total_rps=10_000):
    hit = hit_rate_percent / 100
    avg_latency = hit * cache_ms + (1 - hit) * db_ms
    db_qps = int(total_rps * (1 - hit))
    print(f"{hit_rate_percent:>3}% hits: avg {avg_latency:5.1f} ms, DB load {db_qps:>6,}/s")

for rate in (0, 50, 80, 90, 95, 99):
    simulate(rate)

Even modest caching helps a lot, and the last few percent matter most: 90% → 99% cuts database load another 10×.

The freshness trade-off

Every cache trades speed for freshness. A cached copy is fast but may be stale: the database might have changed a second ago while the cache still holds the old value. For most read-heavy data that's fine:

• A product description cached for 5 minutes: fine, descriptions rarely change.
• A notification count cached for 60 seconds: tolerable, a short delay is acceptable.
• A stock price cached for 5 minutes: dangerous, financial data must be real-time.

So the question is never "should we cache?" but "how stale can this data be?" Different data has different tolerances, and the rest of this caching series is about managing that trade-off: key design, read and write strategies, consistency, invalidation and TTLs, eviction, distribution, high availability, warming, failure modes, security and monitoring and caching in interviews.

The four caching layers

"Closer to the requester" can mean several places. A request can pass up to four caching layers before reaching the database, and each layer can short-circuit the trip. The layers nearest the user are the fastest but the hardest to invalidate; those nearest the database are the freshest but add the most latency. Knowing which layer to use, and which to skip, is a core design skill.

1. Browser cache

The browser keeps HTTP responses on the user's device and checks them before making any network request, so a hit costs zero network latency. The server controls it with HTTP headers: Cache-Control: max-age=3600 tells the browser to reuse a response for an hour without asking the server again.

• Great for static assets: JavaScript bundles, CSS, images and fonts change only on deploy. Put a content hash in the file name (app.a3f2b1.js) and you can cache for a year: a new deploy produces a new file name.
• Tricky for dynamic data: notification counts change every few seconds and carts change on every add. Short TTLs or ETag revalidation help: the browser asks "has this changed?" with If-None-Match, and the server answers with a tiny 304 Not Modified if not, saving the body but not the round trip. For rapidly changing state, requests usually go deeper.

More in browser caching.

2. CDN / edge cache

A CDN keeps content on servers spread across regions. A user in Tokyo asking for an image gets it from a Tokyo edge server rather than the origin in Virginia: roughly 150 ms becomes under 10 ms. CDNs cache at the HTTP layer: on a miss the edge fetches from the origin, stores the response, and serves later requests itself, following the same Cache-Control headers browsers use.

• Excellent for static content at scale: images, video, downloads, pre-rendered pages, and absorbing traffic spikes when hit rates are high.
• Weak for personalized or fast-changing content: a page with the user's name, cart or live prices can't be cached at the edge without careful partitioning. See CDN vs application cache.

3. Application cache

An in-memory store such as Redis or Memcached between the application servers and the database. This is the layer system design interviews care about most. The app checks it first: a hit is sub-millisecond to a few milliseconds over the network; a miss queries the database, stores the result and returns it.

It handles what CDNs can't: dynamic, personalized data with complex access patterns: user profiles, product details, sessions, feed timelines, search results. It's also the layer you control most precisely (what to cache, when it expires, how to invalidate on writes), which brings the complexity covered in the rest of this series.

4. Database buffers

Databases cache too. A relational database keeps frequently used pages in a buffer pool in RAM (a read from it takes about 0.1 ms versus about 8 ms for a disk seek). The database manages it automatically, so you rarely configure it in an interview, but it explains two things: a database whose working set fits in RAM is dramatically faster, and a freshly restarted (cold) database is slow until its buffer pool fills, which is why cache failures that dump traffic onto the database cause cascading slowdowns.

Layers compose

In production the layers work together. A typical request:

1. The browser checks its cache: miss (first visit).
2. The CDN edge checks its cache: hit for static assets, miss for personalized API calls.
3. The application server checks its in-memory cache: hit for popular data.
4. Only if everything missed, the database runs the query, from its buffer pool or disk.

Each layer shields the ones below: the browser absorbs repeat visits, the CDN absorbs geography, the application cache absorbs hot data, and the database handles only cold or freshly written data. The effect is multiplicative: if the browser catches 30% of requests, the CDN 40% of the rest and the application cache 90% of what's left, the database sees only 4.2% of the original traffic.

Choosing the right layer

A worked example

An e-commerce site's product pages query the database for details, price and stock. The database takes 1,000 requests a second, mostly for the same 100 popular products. Each query takes 50 ms, pages feel slow, database CPU sits at 90%, and adding database servers is expensive, yet most queries return the same data as a few minutes ago.

Cache the products in memory and serve repeats from the cache: response time drops from about 50 ms to about 1 ms and database load falls by around 80%. Use caching like this when reads far outnumber writes, when some staleness is acceptable (a price a few minutes old on the product page; checked fresh at checkout), and when you need to cut database load and latency.

This read-through-the-app approach is cache-aside. Alternatives include write-through (writes update the cache and database together) and write-behind (writes update the cache first and the database asynchronously); see cache strategies. Because memory is limited, an eviction policy decides what to drop when the cache is full: LRU removes what hasn't been used recently, LFU removes what's used rarely, and TTL removes entries when they expire; see eviction. Invalidation keeps data fresh: a 5-minute TTL for prices, plus deleting a product's entry immediately when its price changes; see invalidation. Longer TTLs mean staler data but less database load; shorter TTLs mean fresher data and more database hits.

What to cache, and what not to

• Read-heavy, rarely changing data: product details, user profiles, configuration, permission sets.
• Expensive computations: rendered pages, aggregates, recommendation results, search results for popular queries.
• Remote calls: third-party API responses (within their terms), auth token verification keys.
• Not: data that must be exact at read time (balances at checkout, stock at purchase); data read once (churn with no hits); highly personalized data with no reuse; secrets and sensitive data (see cache security).

Common implementations

Treat the cache as an optimization, not a source of truth. The system must still be correct (just slower) if the cache is empty or unavailable, and the database must survive a cold cache for at least a short time. See cache warming and cache high availability.