Latency vs Throughput

Latency is how long one operation takes, measured in time (milliseconds or microseconds). Throughput is how many operations a system completes per unit of time (requests per second, MB per second). They are related but distinct: a highway's latency is the drive time, its throughput is cars per hour. You can widen the highway to carry more cars without making any single trip faster.

Both are core non-functional requirements. High latency makes a system feel slow and hurts user satisfaction, and in real-time products such as video streaming, online games or high-frequency trading even a few extra milliseconds can be damaging. Throughput decides how much load the system can carry as demand grows.

What latency is made of

In the simplest terms latency is a delay: the time from when a request is sent until the response arrives. Seen from the client, that total (also called response time) has two parts:

• Network latency: the time for the request to travel from the client to the server and for the response to travel back, the time spent "on the wire". It depends on physical distance, the number of hops (routers, proxies) the data passes through, and the speed of each link.
• Server-side latency: the time the server takes to understand the request, do the work (calculations, database reads, calls to other services) and build the response. It depends on hardware, how efficient the software is, and how complex the request is.

Managing latency means working on both: getting closer to users and fewer hops for the network part, faster code, queries and caches for the server part.

Measure latency with percentiles, not averages

Latency is never one number. Most requests are fast; a few are slow because of garbage-collection pauses, cache misses, a slow disk or a retry. An average hides those slow requests, so latency is measured in percentiles:

• p50 (median): the slowest of the fastest 50% of requests. Half of requests finish within this time; it's the typical experience.
• p95 / p99: 95% or 99% of requests finish within this time. This is what your unluckiest users see, and they are often your most active ones.
• p99.9 and max: the long tail, where timeouts and retries live.

By definition p99 is always greater than or equal to p50. A real example from a database service (Firestore on Google Cloud) at one moment:

Tail latency

Tail latency is the latency of the slowest requests: the 99th, 99.9th or even 99.99th percentile. Sort the latencies of millions of operations from fastest to slowest and the tail is the long stretch of slow ones at the end.

It matters because it matches the worst user experiences. An average describes general performance but hides the occasional slowdowns people actually notice, so engineers work to cut the tail, not just the average. That is often hard and expensive: it can mean tuning server software, upgrading hardware, changing the architecture, improving load balancing, or adding failover for peaks and outages. Weigh the effort against the benefit.

Tail latency matters more as systems grow. One page often fans out to many backend calls in parallel, and it is only as fast as the slowest of them. If each backend has a 1% chance of being slow, a page that calls 100 backends hits at least one slow call about 63% of the time (1 − 0.99^100). At scale, every user sees your p99.

Tactics that cut tail latency: timeouts with a fallback (show the page without ads), hedged requests (send a second copy to another replica if the first hasn't answered by the p95), avoiding head-of-line blocking, and keeping garbage-collection pauses short. See retries and timeouts.

Reducing latency

1. Use a CDN. A content delivery network keeps copies of content in many locations worldwide, so data travels a shorter distance.
2. Load balance. Spreading traffic over many servers prevents congestion on any one of them, keeping response times low. See load balancing.
3. Cache. Keeping frequently used data in memory, close to where it's needed, cuts the time to fetch it. See caching.
4. Pick the right data center location. Put servers near most of your users: for users in the US, a data center in San Francisco beats one in Singapore.
5. Optimize the server path. Add indexes, remove unnecessary calls, and parallelize independent work. See database optimization.

Throughput and how to raise it

Throughput is the amount of data or number of transactions a system processes in a given period. It directly measures performance and efficiency: higher throughput means the system handles more load, and it decides how well the system copes as demand rises. There are two ways to raise it:

• Reduce the time per request (lower latency), so each worker finishes more requests per second.
• Process more requests at once (more concurrency), by adding parallel capacity.

Concurrency is usually where the big gains are. Three standard moves:

Scale reads with replication

Copy the data onto several servers so many reads can run at the same time. The primary stays the source of truth and sends updates to the replicas; read requests are spread across the replicas, so the system keeps answering quickly under heavy read load. See replication.

Scale writes with partitioning (sharding)

Split the database into smaller parts, each on its own server. Write requests are spread across the partitions, so no single server takes the whole write load and total write throughput grows with the number of partitions. See sharding.

Scale data flow with queues and consumers

A message queue sits between producers and the processes that do the work. Requests are handled asynchronously, and consumers can be scaled out to add processing capacity. This works best when requests can be processed independently of each other.

Little's Law ties them together

For any stable system, concurrency = throughput × latency (L = λW). This simple law is very useful for sizing:

• A service handling 2,000 requests/s with 50 ms latency has on average 2,000 × 0.05 = 100 requests in flight. Size thread pools and connection pools around that.
• If latency degrades to 500 ms at the same throughput, in-flight requests jump to 1,000. That is how a slow database exhausts connection pools and turns a slowdown into an outage.

The throughput–latency trade-off

Latency stays flat while a system is lightly loaded, then rises sharply as utilization approaches 100%, because requests start queuing. Running a server at 95% CPU maximizes throughput per dollar but makes latency unpredictable. Most latency-sensitive services target 50–70% utilization at peak to leave headroom.

Latency numbers worth knowing

The ratios matter more than the exact values: memory is about 1,000× faster than SSD random reads, and a cross-region round trip costs as much as hundreds of in-region calls. More in back-of-the-envelope estimation and caching.