Hot Keys and Hot Partitions: Detecting and Fixing Skew

A hot key is a single key that receives far more traffic than others, and a hot partition is a shard or partition that carries a disproportionate share of load. Sharding assumes load spreads evenly across partitions; when it doesn't, one node saturates while the rest sit idle, and adding nodes doesn't help. A celebrity's profile, a viral post, a flash-sale product or a huge tenant can each turn a well-sharded system into a single-server bottleneck.

Where skew comes from

• Popular entities: a celebrity account, a trending video, a popular product during a sale.
• Poor key choice: sharding by date sends all of today's writes to one partition; sharding by country sends most traffic to the largest country.
• Monotonic keys: auto-increment IDs or timestamps with range partitioning concentrate new writes on the last range.
• Big tenants in a multi-tenant system. See multi-tenancy.
• Default values: a null or "unknown" key that many rows share, which also causes skew in Spark joins.

Detecting hot keys

• Per-partition metrics: request rate, CPU, latency and throttling per shard. A single shard far above the median is the signal.
• Top-k key tracking: sample requests or use a count-min sketch to find the heaviest keys without counting everything. See probabilistic data structures.
• Managed databases often report hot partitions or throttle them, which shows up as errors on just some keys.

Fixes for read-hot keys

• Cache aggressively: a hot key is the ideal cache entry. Put it in an in-process cache or CDN, not only a shared cache, because a single cache node can itself become hot. See caching.
• Replicate the hot key: keep copies under several cache keys (post:42#1 ... post:42#8) and pick one at random per read, spreading load across nodes.
• Request coalescing: collapse concurrent misses for the same key into one backend request. See cache stampedes.
• Read replicas for the partition that owns the key.

Fixes for write-hot keys

• Key salting (write sharding): split the key into N sub-keys, for example likes:post42:0 to likes:post42:15, write to a random one, and sum them on read. Writes spread across partitions at the cost of a fan-in read.
• Buffer and batch: accumulate increments in memory or a queue and apply them in batches, so a viral post's like count is updated every second rather than per like.
• Change the key: use a hash of a high-cardinality attribute, or compose keys (tenant_id + user_id) instead of a low-cardinality one.
• Avoid monotonic range keys: hash them, or prefix with a bucket number, so new writes don't all land on the last range.
• Split or isolate: move a huge tenant or entity to a dedicated partition or cell.

Design-time prevention

• Ask about the distribution of keys, not just the total volume, when choosing a shard key.
• Plan for the celebrity case explicitly, as in feed fan-out.
• Use consistent hashing with virtual nodes so partitions can be rebalanced at a fine grain.
• Rate-limit per key or per tenant so one entity can't consume a shared partition's capacity. See rate limiting.