Tradeoffs Have Names: CAP, Latency, Cost, Headroom

Why this matters

CAP is the most quoted and least useful thing in this field, because the version most people carry around — pick two of three — is not what the theorem says and does not describe any decision you will actually make. Partitions are not something you choose; they are something the network does to you, so the real choice is only what to do during one, and that choice comes up a few times a year. The choices you make every single day are the ones CAP does not mention: whether to answer this read from the nearest possibly-stale copy or pay a coordination round trip to be certain, whether to run one copy or three, whether to buy 30 percent headroom or 300. Naming a tradeoff precisely is what turns it from an argument into a decision, because once it has a name it has a price, and once it has a price you can ask whether it is worth paying. This task gives you names and prices for the four you will meet constantly.

Demo

CAP, corrected: PACELC. If there is a Partition, choose Availability or Consistency. Else — meaning on every ordinary day, which is nearly all of them — choose Latency or Consistency. The else branch is the one you live in, and it is missing from the popular version of CAP entirely. DynamoDB and Cassandra are PA/EL systems: available during partitions, fast the rest of the time, and you accept staleness in both cases. Spanner is PC/EC: it refuses rather than diverge, and it pays a coordination round trip on every strongly consistent read for the rest of its life. Neither is better. They have different prices.

Latency versus consistency, with a real price tag. In DynamoDB a strongly consistent read costs exactly twice as much as an eventually consistent one — one read capacity unit versus half of one — and it must go to the leader replica, so it is slower and it is unavailable if that leader is unreachable. That is not a metaphor; it is the rate card. At the photo app's 104 reads per second the difference is a few dollars a month and you should simply buy strong consistency everywhere. At 100x that same choice is a few hundred dollars a month plus a latency and availability penalty on every read, and now it deserves a conversation. The same tradeoff is free at one scale and load-bearing at another, which is why estimation comes before tradeoffs.

Cost versus redundancy, priced as multipliers. One availability zone is 1x and dies with the zone. Multi-AZ with a synchronous standby is roughly 2x the database bill and survives a zone. Multi-region active-passive is roughly 2x again plus cross-region transfer and a promotion procedure you must actually rehearse. Multi-region active-active is 2x plus conflict resolution, which is a permanent tax on your engineering time rather than your invoice. Say the multiplier out loud. A design that quietly assumes 4x infrastructure spend for a business that cannot afford 2x is not a design.

Simplicity versus headroom. Every component you add to survive a future 10x is a component you operate today, on a four-person team, at 3 a.m. The honest question is not can this scale, it is what will this cost me every week between now and the day it needs to. A single Postgres box with a good backup story and 5x headroom beats a five-service event-driven architecture with 500x headroom for almost every system that exists, right up until the day it does not — and the design skill is knowing roughly when that day arrives, and what the migration looks like.

Try it yourself

1. Run the script and find the read rps at which the strong-consistency premium crosses 100 US dollars a month. Then decide, with a reason, whether you would pay it for a wallet balance and whether you would pay it for a follower count.
2. For a product you know, sort five different reads into eventually-consistent and strongly-consistent buckets. If everything lands in one bucket you have not looked hard enough — find the read that breaks the pattern.
3. Write the redundancy multiplier for your current setup: 1x single zone, about 2x multi-AZ, about 4x multi-region. Then compute what that multiplier costs annually in both currencies and express it as a fraction of one full-time salary.
4. Verify the W + R greater than RF rule by hand for RF=3: try ONE/ONE, ONE/QUORUM, QUORUM/QUORUM and ALL/ONE. For each, say which replica failures it survives and whether a read can miss a committed write.
5. Take one system you consider over-engineered and one you consider under-engineered, and for each name the specific tradeoff that was mispriced. Vague answers here mean you have not actually found the decision yet.

References

• Abadi — Consistency tradeoffs in modern distributed design (PACELC)
• Brewer — CAP twelve years later
• Dynamo — Amazon's highly available key-value store (SOSP 2007)
• Spanner — Google's globally distributed database (OSDI 2012)