Backpressure and Flow Control
Backpressure explained: bounded queues, flow control, dropping vs blocking vs buffering, and keeping fast producers from sinking slow consumers.
Backpressure is how a system tells upstream components to slow down when downstream components can't keep up. Without it, a fast producer fills buffers until memory runs out, latency climbs without limit, or the slow component falls over. With it, overload is pushed back to where it can be handled: by waiting, by dropping low-value work, or by rejecting requests early.
The problem: unbounded buffers
Every queue, buffer and thread pool between two components hides a rate mismatch, until it doesn't. If a producer emits 10,000 events/s and a consumer handles 8,000/s, the backlog grows by 2,000 every second: 7.2 million after an hour. Memory grows, every item waits behind millions of others, and when the process finally crashes the buffered work is lost too.
Strategies when downstream is full
| Strategy | What happens | Good for |
|---|---|---|
| Block / pull-based flow control | The producer waits until there's room; consumers only request what they can handle | Pipelines where every item matters (file processing, ETL) |
| Drop | Discard new (or oldest) items when full | Telemetry, metrics, live video frames, location pings |
| Sample / aggregate | Keep a representative subset or combine items | Analytics events, logs at extreme volume |
| Reject early | Return 429 or 503 at the edge with Retry-After | User-facing APIs: fail fast instead of timing out slowly. See rate limiting |
| Buffer durably | Spill to a durable queue or log and catch up later | Spiky but finite bursts (flash sales). See message queues |
| Scale out | Add consumers when lag grows | Elastic workloads; still needs a limit while scaling happens |
Backpressure in practice
- TCP has backpressure built in: the receive window shrinks when the receiver's buffer fills, and the sender slows. See TCP vs UDP.
- Pull-based consumers (Kafka, reactive streams) are naturally backpressured: consumers fetch the next batch only when ready. Lag accumulates in the durable log instead of in memory.
- Bounded thread pools and queues in services: when the request queue is full, reject immediately rather than queuing for minutes.
- Connection pools cap concurrent database calls; requests waiting for a connection should time out quickly.
- gRPC and HTTP/2 have per-stream flow control windows.
Queues don't remove overload
Putting a queue in front of a slow service converts overload into latency. That's useful for a short burst, but if the average input rate exceeds the processing rate, the queue grows until something breaks. Always pair a queue with monitoring of depth and oldest-message age, autoscaling on lag, and a plan for when the backlog can't be cleared (shed the lowest-priority work). See load shedding.
Prefer failing fast to failing slow. A request rejected in 5 ms can be retried elsewhere or shown as "try again"; a request that times out after 30 seconds has already tied up threads, connections and the user's patience.
Little's Law and queue sizing
In-flight work = arrival rate × time in system. If you want at most 500 ms of queueing at 2,000 requests/s, the queue should hold at most about 1,000 requests; anything larger only adds latency nobody will wait for. See latency vs throughput.
Where it’s used
Downstream stages signal demand, so upstream stages emit only as many elements as requested.
Pull at their own pace; growing consumer lag signals the need to scale, without exhausting broker memory.
Adaptive concurrency limiters reduce allowed in-flight requests when latency rises, rejecting excess load early.
Agents like Fluent Bit use bounded buffers and drop or spill to disk under pressure instead of starving the application of memory.
In an interview
- Whenever two components run at different speeds, say what bounds the buffer between them and what happens when it fills.
- Choose per data type: block for must-not-lose work, drop or sample for telemetry, reject early for user requests.
- Monitor queue depth and lag, autoscale consumers on them, and plan for a backlog you can't clear.
- Prefer fast rejections (429/503 with Retry-After) over slow timeouts.