Demystifying Latency vs Throughput vs Bandwidth: Capacity, Queuing & Little’s Law

Demystifying Latency vs Throughput vs Bandwidth: Capacity, Queuing & Little’s Law

1. The Three Fundamental Performance Metrics

In system design and distributed networking, developers often conflate these three concepts:

+-------------+-------------------------------------------------------------+
| Metric      | Physical Analogy (Highway System)                          |
+-------------+-------------------------------------------------------------+
| Bandwidth   | Number of lanes on the highway (Max theoretical capacity)   |
| Latency     | Time required for a single car to travel from point A to B  |
| Throughput  | Number of cars actually passing the toll booth per minute   |
+-------------+-------------------------------------------------------------+

A network connection with high bandwidth can still have terrible latency (e.g., satellite links).

2. Little's Law in System Capacity

L = λ * W

Where:
L = Average number of concurrent requests in the system (Concurrency)
λ = Throughput (Requests per second)
W = Average Latency (Response time in seconds)

Example Calculation:

If your API processes requests in $W = 200\text{ms}$ ($0.2\text{s}$) and you want a throughput of $\lambda = 5,000\text{ RPS}$: $$\text{Required Concurrent Connections } L = 5000 \times 0.2 = 1,000\text{ connections}.$$ If your database connection pool is limited to 100 connections, requests will queue, latency will spike, and the system will saturate!

3. The Saturation Cliff (Queuing Delay)

Latency
  ^
  |                                        / (Saturation Cliff)
  |                                       /
  |                                      /
  |-------------------------------------/
  +---------------------------------------------> Throughput / Load

As resource utilization approaches 100%, queue wait times grow non-linearly toward infinity.

4. Practical Engineering Guidelines

1. Reduce Latency First: Halving your API latency from 200ms to 100ms automatically doubles your system's throughput capacity under the same connection pool limit.
2. Set Resource Limits: Reject requests early (HTTP 429 / 503) instead of letting requests queue into multi-second latency spirals.