Latency, Throughput & Percentiles

By the end of this lesson you will be able to explain the difference between latency and throughput, read p50 and p99 numbers like an engineer, see why a page that calls many services gets slow, and use Little's Law to size a system.

Latency and throughput are different things

Latency is how long one request takes, from start to finish. It is measured in time: milliseconds (ms) or microseconds.

Throughput is how many requests the system finishes per unit of time. It is measured in requests per second, messages per second or MB per second.

Think of the Mumbai dabbawalas. The time from a kitchen in Andheri to an office desk in Churchgate is the latency of one tiffin, a few hours. The number of tiffins delivered every day across the city is the throughput, in lakhs. Adding more dabbawalas raises throughput. It does not make the local train run faster, so latency stays the same.

Systems behave the same way:

• Adding servers usually raises throughput.
• Reducing latency needs different tricks: caching, fewer round trips, faster storage, moving closer to users.
• Batching (sending 100 messages together) often raises throughput but adds latency for each message, because messages wait for the batch to fill.

Rough numbers every designer should know

You do not need exact values. You need a feeling for which operations are cheap and which are expensive. These are rough orders of magnitude; real hardware varies.

The ratios matter more than the numbers. Reading memory is hundreds of times faster than reading an SSD, and a cross-continent round trip is hundreds of times slower than a round trip inside one data centre. This is why caches keep hot data in memory, and why you avoid calling a far-away service inside a loop.

Averages hide pain: use percentiles

Suppose 100 requests take 20 ms each, and 1 request takes 3 seconds. The average looks fine, but one user waited 3 seconds and may never come back.

A percentile answers "what latency do X% of requests beat?". The p50 (median) is the typical request. The p99 is the latency that 99% of requests beat; only the slowest 1% are worse. Slow outliers are called tail latency.

Step 1: run the code to generate 10,000 fake latencies. Most are fast, but 2% hit a slow path such as a cache miss or a garbage-collection pause.

import random

rng = random.Random(42)
latencies = []
for _ in range(10_000):
    ms = rng.lognormvariate(3.0, 0.4)          # typical requests, around 20 ms
    if rng.random() < 0.02:
        ms += rng.uniform(200, 800)            # 2% hit a slow path
    latencies.append(ms)

def percentile(values, p):
    ordered = sorted(values)
    rank = max(1, round(p / 100 * len(ordered)))   # nearest-rank method
    return ordered[rank - 1]

print(f"average: {sum(latencies) / len(latencies):6.1f} ms")
for p in [50, 90, 95, 99, 99.9]:
    print(f"p{p:<5}: {percentile(latencies, p):6.1f} ms")

average:   32.7 ms
p50   :   20.4 ms
p90   :   35.2 ms
p95   :   42.8 ms
p99   :  554.5 ms
p99.9 :  804.2 ms

Step 2: compare the lines. The average (about 33 ms) describes nobody: typical users see about 20 ms, while the unlucky 1% wait more than half a second. Service targets are therefore written as percentiles, for example "p99 under 300 ms". You will use these in Observability.

Fan-out makes the tail everyone's problem

Modern pages rarely call one service. A product page might call price, stock, reviews, offers and delivery-estimate services in parallel, and must wait for the slowest one. This is called fan-out.

If each service is fast 99% of the time, what is the chance that all of them are fast? It is 0.99 multiplied by itself once per call.

for calls in [1, 5, 10, 50, 100]:
    all_fast = 0.99 ** calls
    print(f"{calls:>3} parallel calls: {all_fast:6.1%} of pages are fast, "
          f"{1 - all_fast:6.1%} hit at least one slow call")

  1 parallel calls:  99.0% of pages are fast,   1.0% hit at least one slow call
  5 parallel calls:  95.1% of pages are fast,   4.9% hit at least one slow call
 10 parallel calls:  90.4% of pages are fast,   9.6% hit at least one slow call
 50 parallel calls:  60.5% of pages are fast,  39.5% hit at least one slow call
100 parallel calls:  36.6% of pages are fast,  63.4% hit at least one slow call

With 100 calls, a "rare" 1% slowness hits almost two pages out of three. Google engineers described this effect in the paper "The Tail at Scale" (2013). Common fixes:

• Fewer calls per page: combine data, or precompute it.
• Timeouts with fallbacks: if reviews are slow, show the page without them.
• Hedged requests: if a reply is late, send the same read to a second replica and use whichever answers first. This costs a little extra load.

Little's Law: connecting the two

Little's Law links throughput, latency and the number of requests in progress:

requests in progress (L) = arrival rate (lambda) x time each request spends (W)

It is simple but very useful. Think of a railway ticket hall. If 2 people arrive per minute and each spends 10 minutes inside, there are on average 20 people in the hall.

For servers, it tells you how many requests are "in flight" at once, which decides how many worker threads, database connections or open sockets you need.

def in_flight(rate_per_sec, latency_ms):
    return rate_per_sec * latency_ms / 1000

print("Payment API, 1,200 req/s at 150 ms:", in_flight(1200, 150), "requests in flight")
print("Same API when the bank is slow, 1,500 ms:", in_flight(1200, 1500), "requests in flight")

pool_size = 200
for latency in [150, 500, 1500]:
    needed = in_flight(1200, latency)
    status = "OK" if needed <= pool_size else "pool exhausted, requests queue up"
    print(f"latency {latency:>4} ms needs {needed:6.0f} connections: {status}")

Payment API, 1,200 req/s at 150 ms: 180.0 requests in flight
Same API when the bank is slow, 1,500 ms: 1800.0 requests in flight
latency  150 ms needs    180 connections: OK
latency  500 ms needs    600 connections: pool exhausted, requests queue up
latency 1500 ms needs   1800 connections: pool exhausted, requests queue up

This explains a famous failure pattern. A downstream system gets slow, latency rises ten times, and suddenly ten times more requests are stuck waiting. Threads and connections run out, and the healthy service also stops answering. Timeouts and circuit breakers, covered in Reliability, exist to stop this chain reaction.

Common mistakes

• Reporting only averages. Always look at p95 and p99 as well.
• Assuming more servers fix latency. More servers raise throughput; latency needs fewer or faster steps.
• Sequential calls that could be parallel. Three 100 ms calls in a row take 300 ms; in parallel they take about 100 ms.
• No timeouts. One slow dependency can freeze every thread you have.

Key points

• Latency is time per request; throughput is requests per second.
• Memory is far faster than disk or network; long-distance round trips are the slowest step of all.
• Use percentiles (p50, p95, p99) because averages hide the slow tail.
• Fan-out to many services makes tail latency common, so limit calls and use timeouts.
• Little's Law, L = lambda x W, tells you how many requests are in flight and how big pools must be.

rate = 800
pool = 100
for latency_ms in [60, 900]:
    in_flight = rate * latency_ms / 1000
    ok = "enough" if in_flight <= pool else "not enough"
    print(f"{latency_ms} ms -> {in_flight:.0f} in flight, pool of {pool} is {ok}")

60 ms -> 48 in flight, pool of 100 is enough
900 ms -> 720 in flight, pool of 100 is not enough