Understanding network and system performance metrics: why high bandwidth does not guarantee low latency, Little’s Law, and saturation bottlenecks.
In system design and distributed networking, developers often conflate these three concepts:
+-------------+-------------------------------------------------------------+
| Metric | Physical Analogy (Highway System) |
+-------------+-------------------------------------------------------------+
| Bandwidth | Number of lanes on the highway (Max theoretical capacity) |
| Latency | Time required for a single car to travel from point A to B |
| Throughput | Number of cars actually passing the toll booth per minute |
+-------------+-------------------------------------------------------------+
A network connection with high bandwidth can still have terrible latency (e.g., satellite links).
L = λ * W
Where:
L = Average number of concurrent requests in the system (Concurrency)
λ = Throughput (Requests per second)
W = Average Latency (Response time in seconds)
If your API processes requests in $W = 200\text{ms}$ ($0.2\text{s}$) and you want a throughput of $\lambda = 5,000\text{ RPS}$: $$\text{Required Concurrent Connections } L = 5000 \times 0.2 = 1,000\text{ connections}.$$ If your database connection pool is limited to 100 connections, requests will queue, latency will spike, and the system will saturate!
Latency
^
| / (Saturation Cliff)
| /
| /
|-------------------------------------/
+---------------------------------------------> Throughput / Load
As resource utilization approaches 100%, queue wait times grow non-linearly toward infinity.