Component Vocabulary: What Each Box Buys and Costs

Why this matters

Every component in a distributed system is a trade, and the trade is never free. A cache buys latency and pays in staleness. A queue buys decoupling and pays in eventual, out-of-order, possibly duplicated delivery. A read replica buys read throughput and pays in replication lag, which means a user can write something and then not see it. A blob store buys effectively unlimited cheap bytes and pays by making those bytes unqueryable and expensive to move. If you can name the property a component buys but not the currency it charges, you do not understand the component — you have memorised its logo. Worse, the costs compound in ways that surprise people: a cache with a 90 percent hit rate silently becomes load-bearing infrastructure, because the day it dies your database gets ten times its normal traffic in a single second. That is not a caching problem; it is a capacity problem you created when you added the cache and did not write down what it cost.

Demo

Trace one request through the standard chain and, at every hop, say the two things: what it buys, what it charges. The diagram below is that chain. Read the second and third lines of every box — they are the part people skip.

Then do the arithmetic that makes the cache trade concrete. Assume a cache hit costs 1 ms and a database read costs 40 ms, the service takes 2,000 reads per second, and the database can sustain 400.

At a 90 percent hit rate the mean latency is 4.9 ms, which looks excellent. But the 99th percentile is still 40 ms, because at a 90 percent hit rate one request in ten misses and the 99th-percentile request is definitively a miss. A cache moves the mean long before it moves the tail. The tail only improves once the hit rate crosses 99 percent, and getting from 90 to 99 is far harder than getting from 0 to 90.

Now the part that ends up in postmortems. At a 90 percent hit rate the database sees 200 rps — half its capacity, comfortable. If the cache goes away, the database sees 2,000 rps instantly: a 10x jump to five times its capacity, which is not a slowdown but an outage. And the better your cache is, the worse this gets: at a 99 percent hit rate the database normally sees 20 rps, so cache loss is a 100x step change. The cost of a cache is not staleness. The cost of a cache is that you now have a component whose failure mode is a 100x traffic spike on the thing behind it.

Every box in the diagram has an equivalent second-order cost. A load balancer buys one address in front of many servers and charges you a new thing to operate, plus health-check semantics that decide what down means. A queue buys the right to accept work faster than you can do it, and charges you the entire class of bugs that begins with the words but the message arrived twice. A search index buys queries your database cannot answer, and charges you a second copy of your data that drifts from the first.

Try it yourself

1. Run the script and find the lowest hit rate at which p99 improves at all. Confirm it is 0.99, then write one sentence explaining why the mean improved 8x at a 90 percent hit rate while the tail did not move a millisecond.
2. Raise DB_CAPACITY_RPS until cache death at a 99 percent hit rate is survivable. Note that you must provision for 2,000 rps — the full uncached load — which means the cache saved you latency but saved you no capacity at all.
3. Add a warm-up model: assume that after cache loss the hit rate recovers linearly over 60 seconds, and compute how many requests the database sees in that minute. Compare it to what the database serves in a normal minute.
4. For each of the nine boxes in the diagram, write the sentence that would let you delete it from a design. If you cannot write a deletion sentence for a box, you also cannot justify adding it.
5. Find one component in a system you work on whose failure would cause a step change in load somewhere else, and write down the multiplier. Anything above 5x is an incident waiting for a bad Tuesday.

References

• AWS Builders Library — Caching challenges and strategies
• Scaling Memcache at Facebook (NSDI 2013)
• Kafka design documentation
• AWS Builders Library — Avoiding fallback in distributed systems