Cache High Availability: Replicas, Sentinel and Redis Cluster

A sharded cache spreads keys over several nodes, but if one node crashes, every key on it becomes a miss at once. If it held popular keys, the database suddenly takes their full load. High availability keeps the cache serving when individual nodes fail, through replication and automatic failover.

Replication: primary and replicas

Each cache node (the primary) has one or more replicas that mirror its data. The primary takes all writes and replicates them asynchronously to the replicas; reads can be served by the primary or any replica.

• Fast recovery: when the primary fails, a replica already holds a nearly complete copy in memory, so promoting it is quick. Only writes that reached the primary after the last replication batch are lost, typically milliseconds of data.
• Read scaling: replicas add read capacity. A primary with 2 replicas handles roughly 3× the reads of one node. Read from replicas for eventually consistent data and from the primary when you need the latest value.
• Replication lag: because replication is asynchronous, replicas can trail the primary by milliseconds to seconds under load. For caches that's usually fine, since a cache is already eventually consistent.

Automatic failover with Redis Sentinel

When a primary dies, something must notice and promote a replica. Redis Sentinel automates it: a separate process, run as 3 or more instances for a quorum, that monitors the primaries and replicas. When a majority of Sentinels agree a primary is unreachable, they start a failover:

1. Pick the replica with the most up-to-date data.
2. Promote it to primary.
3. Point the other replicas at the new primary.
4. Tell client libraries about the new topology.

Failover takes a few seconds, during which writes to the failed primary are lost and reads may fail. Client libraries that support Sentinel then reconnect to the new primary automatically.

Split-brain risk. If the old primary wasn't actually dead, just cut off by a network partition, both it and the new primary may accept writes. When the partition heals, the old primary finds it has been demoted and discards its writes, so anything written to it during the partition is lost. For a cache that's acceptable, because the data can be re-fetched from the database; it's exactly why a database needs stronger guarantees. See leader election.

Redis Cluster: built-in HA

Redis Cluster combines sharding and replication. Each shard (a range of hash slots) has a primary and one or more replicas, and when a shard's primary fails the cluster promotes a replica itself; no separate Sentinel is needed. The cluster stays available as long as every shard has at least one surviving node. If an entire shard (primary and all replicas) is lost, only that shard's keys become unavailable.

The cluster-require-full-coverage setting decides what happens when a whole shard is down. With yes (the default) the cluster stops serving key operations (reads and writes) and reports CLUSTERDOWN. With no, the surviving shards keep serving their keys and only the failed shard's keys return errors, usually the better choice for a cache.

The cache as a non-critical layer

A cache differs from a database in one crucial way: the database is the source of truth. If the cache fails completely, the application can query the database directly. It will be slower, but nothing is lost. That changes the HA calculation:

• For a database, losing a node risks losing data, so it may need stronger durability (sometimes synchronous replication), depending on its recovery point objective (RPO). See disaster recovery.
• For a cache, losing a node risks only slower responses, so asynchronous replication (faster, simpler) is acceptable: the worst case is re-fetching from the database.

Degrade gracefully. When a cache node fails, the application must handle the burst of misses without a cascade: circuit breakers stop the database being overwhelmed by a miss storm, and rate limits on the miss path keep database load within capacity. The cache re-warms naturally as requests flow; see cache warming.

from pybreaker import CircuitBreaker, CircuitBreakerError
db_breaker = CircuitBreaker(fail_max=20, reset_timeout=10)

def get_product(product_id):
    key = f"product:v1:detail:{product_id}"
    try:
        cached = cache.get(key)               # cache down? treat as a miss, don't crash
    except ConnectionError:
        cached = None
    if cached is not None:
        return cached
    try:
        row = db_breaker.call(db.fetch_product, product_id)
    except CircuitBreakerError:
        return {"id": product_id, "unavailable": True}   # protect the DB during a miss storm
    try:
        cache.set(key, row, ex=600)
    except ConnectionError:
        pass
    return row