Performance · foundation
Throughput vs latency
Latency is the elapsed time for an individual operation, while throughput is the number or amount of operations completed per unit time.
Why it matters
A system can process a high aggregate rate while individual requests wait too long, so performance goals and measurements must distinguish rate from response time.
Mental model
How to reason about throughput vs latency
Latency follows one item from arrival to completion. Throughput counts completions across an interval. Queues connect them: load near capacity can sharply raise waiting time before throughput visibly falls.
Analogy
A highway's throughput is cars passing per hour; one car's travel time is latency. More lanes can raise throughput, while a toll queue can make each journey slower.
Examples
See the boundary, not just the happy path
Worked example · Batch for throughput
flush 100 writes together every 50 msAmortizing overhead may raise writes per second, while the earliest write waits for the batch and sees higher latency.
Worked example · Measure a distribution
p50 = 40 ms, p99 = 900 msA median alone hides the slowest one percent; percentiles expose tail latency that affects real users and fan-out services.
Useful contrast · High rate, slow requests
10,000 requests/s with a 3-second queueHigh throughput and high latency can coexist when the system completes many requests after substantial waiting.
Common mistakes
Misconceptions to remove early
Reporting only average latency
Averages can conceal multimodal behavior and severe tails. Report relevant percentiles and the measurement window alongside load.
Load-testing without a demand model
A closed test that waits for each response can reduce offered load when latency rises, hiding queue growth. Model realistic arrivals and concurrency.
Quick check
Can you predict the result?
1. Which metric answers how long one request took?
- • Latency
- • Throughput
- • Cache capacity
2. Why can latency rise steeply as offered load approaches capacity?
Keep building