Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
At a thousand requests a second the tail is a curiosity you look at during incident review. At a million requests a second it is a staffed problem. Your p99.9 — the slice most teams treat as an acceptable loss — is a thousand requests every second, eighty-six million a day, which is more traffic than most companies have in total. Somebody is having that experience continuously, forever, while your dashboard reads healthy, and the reason your dashboard reads healthy is that you are looking at a mean. Means are actively misleading for latency because real latency distributions are not bell-shaped; they are bimodal, with a fast path and a slow path and almost nothing in between. A cache with a 98 percent hit rate produces a mean of 2 ms and a p99 of 50 ms, and the mean is a number that describes exactly zero of your requests. Then fan-out multiplies the damage: a page that waits on twenty backend calls has twenty independent chances to catch a slow one, so a one percent backend slow rate becomes an eighteen percent slow page. And the final trap is measurement itself — averaging the p99 of twenty hosts does not give you the fleet p99, and a load generator that waits for a response before sending the next one stops sending during the exact stall you were trying to measure.
Three moves, in order. Build a deliberately bimodal distribution and watch the mean disagree with every percentile. Convert those percentiles into requests per second, which is the translation that makes them feel real. Then apply fan-out and watch a small backend problem become a large page problem.
# tails.py — why the mean lies at a million requests a second
import random, statistics
random.seed(7)
RPS = 1_000_000
MISS_RATE, HIT_MS, MISS_MS = 0.02, 1.0, 50.0 # a cache with a 98% hit rate
sample = sorted(MISS_MS if random.random() < MISS_RATE else HIT_MS
for _ in range(200_000))
def p(q):
return sample[int(q * len(sample)) - 1]
print(f"mean {statistics.fmean(sample):.2f} ms | p50 {p(.50):.1f} | "
f"p95 {p(.95):.1f} | p99 {p(.99):.1f}")
# mean 1.98 ms | p50 1.0 | p95 1.0 | p99 50.0
# The mean says two milliseconds. One user in fifty waits fifty.
# 1. Translate percentiles into a rate. This is the move nobody makes.
for q, label in ((0.99, "p99"), (0.999, "p99.9"), (0.9999, "p99.99")):
n = RPS * (1 - q)
print(f"{label:>7}: {n:>7,.0f} requests/second slower than this "
f"({n * 86400 / 1e6:,.1f} million/day)")
# p99: 10,000 requests/second slower than this (864.0 million/day)
# p99.9: 1,000 requests/second slower than this (86.4 million/day)
# p99.99: 100 requests/second slower than this (8.6 million/day)
# 2. Fan-out. A page that waits on N backend calls inherits N chances
# of touching the tail.
p_slow = 0.01
for n in (1, 5, 20, 100):
print(f"{n:>3} backend calls -> {1 - (1 - p_slow) ** n:6.1%} of pages "
f"hit at least one p99 call")
# 1 backend calls -> 1.0% of pages hit at least one p99 call
# 5 backend calls -> 4.9% of pages hit at least one p99 call
# 20 backend calls -> 18.2% of pages hit at least one p99 call
# 100 backend calls -> 63.4% of pages hit at least one p99 callMISS_RATE to 0.001 and re-run. The mean barely moves, to about 1.05 ms, but p99.9 is still 50 ms — and at a million requests a second that is a thousand people a second on the slow path, about 86 million a day.wrk -t8 -c400 -d60s and once with wrk2 -t8 -c400 -d60s -R 50000 --latency. Compare the two reported p99 figures. The gap between them is coordinated omission, and the closed-loop number is the flattering one.histogram_quantile over a _bucket series in Prometheus, or a t-digest merge — and record how far apart the two p99 values are.p_slow to 0.001, your backend's p99.9 fraction, with 20 calls: about 2 percent of pages. Say out loud whether 2 percent of page loads being slow is acceptable, because with that fan-out that is the number you are currently shipping.