Load Shedding and Graceful Degradation
Load shedding and graceful degradation: rejecting work early, prioritising traffic, and keeping core features alive under overload.
Load shedding deliberately rejects some work when a system is overloaded, so it can keep serving the rest well. Graceful degradation turns off non-essential features to protect the essential ones. Both accept partial failure on purpose, because the alternative under overload is total failure: every request slows down, times out, and gets retried.
Why overload collapses systems
Past a certain load, throughput doesn't plateau; it drops. Queues grow, latency exceeds client timeouts, clients retry, the extra retries add more load, and the server spends its capacity on requests whose callers have already given up. This congestion collapse is why an overloaded service often serves fewer successful requests than a healthy one at half the traffic.
Load shedding techniques
- Concurrency limits: cap in-flight requests per instance; reject immediately when full instead of queueing. Adaptive limiters lower the cap automatically when latency rises.
- Bounded queues with deadlines: drop requests that have waited longer than the client will wait (if a request queued for 2 s and the client times out at 1 s, processing it is pure waste).
- Prioritisation: classify traffic and shed the least important first: background sync and prefetch before user-initiated reads, reads before checkout.
- Per-client fairness: shed the noisiest clients first so one tenant's spike doesn't hurt everyone. See rate limiting.
- Shed early: reject at the edge or load balancer, before spending work on parsing, auth and database calls.
- Cheap rejections: return 503 or 429 with
Retry-Afterin microseconds.
Graceful degradation
Decide in advance which features are critical and which can be turned off or simplified under stress:
| Normal | Degraded mode |
|---|---|
| Personalised recommendations | Popular items from a cache, or the section hidden |
| Real-time inventory counts | "In stock" from a cached value, verified at checkout |
| Full search with facets and spelling correction | Basic keyword search |
| High-resolution images and video | Lower bitrates |
| Live comment counts and typing indicators | Updated less often or paused |
Implement degradations as feature flags or kill switches that operators (or automation) can flip, and pair dependencies with circuit breakers and fallbacks so degradation also happens automatically when a dependency fails.
Designing for overload
- Know your capacity per instance through load testing, and alert well before it's reached.
- Autoscale, but remember scaling takes minutes; shedding handles the seconds in between. See autoscaling.
- Make clients cooperate: honour Retry-After, back off with jitter, and avoid retrying user actions automatically during an outage. See retries.
- Protect dependencies: shedding at your service is useless if every rejected request still triggered a database query.
- Test it: run game days where you push a service past its limits and verify it degrades instead of collapsing.
Where it’s used
Services classify request criticality and shed lower-criticality traffic first; client-side throttling stops clients sending requests that are likely to be rejected.
Uses admission control and degraded modes so the buy button keeps working on Prime Day even if recommendations or reviews are trimmed.
Prioritises playback traffic over less critical requests and has fallbacks for nearly every personalised component.
Virtual waiting rooms admit users at a controlled rate instead of letting the whole crowd hit checkout at once.
In an interview
- Say what happens when traffic exceeds capacity: which requests are rejected, how, and what users see.
- Prioritise critical paths (checkout, login) and list features you'd degrade first.
- Use concurrency limits and fast 503/429 responses instead of unbounded queues.
- Combine shedding with autoscaling, circuit breakers and client backoff.