From Cycles to Cores: Sizing the Fleet's CPU

Why this matters

The most expensive sizing mistake in this whole subject is using latency to buy CPU. A request that takes 20 ms of wall clock might use 60 microseconds of CPU, because the other 19.94 ms is the socket waiting for a reply that some other machine is producing. Size cores off the wall clock and you conclude you need twenty thousand of them; size off CPU and the honest answer is sixty. That is a factor of three hundred, and it shows up in real capacity plans, in real budget approvals, and in real interview answers. The fix is to price a request the way you would price a bill of materials: enumerate what it does, attach a microsecond cost to each line, and add them up. Once you have that number the fleet falls out of one multiplication, and something else becomes visible that is hard to see any other way — at a million requests a second, twenty microseconds of CPU is worth tens of machines and lakhs of rupees a year. A signature verification you do on every request instead of caching the result is not a code-style question at this scale. It is a purchase order.

Demo

First the reference costs, so the units are grounded. Then one realistic endpoint — three Redis calls, a token verification, a JSON encode — priced line by line, converted into cores, and then re-priced with a single line changed so you can see what one microsecond is worth.

# cores.py — cycles in, cores out
# Reference costs on a modern x86 core. Orders of magnitude, not precision.
NS = {
    "L1 hit":                            1,
    "L2 hit":                            4,
    "L3 hit":                           20,
    "main memory (cache miss)":        100,
    "uncontended mutex lock+unlock":    20,
    "syscall round trip":              500,
    "NVMe 4 KB read":               50_000,
    "same-AZ network round trip":  250_000,
    "Mumbai to N. Virginia round trip": 180_000_000,
}
for k, v in NS.items():
    print(f"{k:<34} {v:>13,} ns")

# One realistic endpoint, priced in CPU microseconds (NOT wall clock).
CPU_US = {
    "TLS record encrypt/decrypt (AES-NI, 2 KB)":  2,
    "HTTP parse and routing":                     4,
    "JWT verify (ES256, per request)":           25,
    "3x Redis: syscalls, memcpy, RESP parse":    18,
    "business logic":                            10,
    "JSON encode, 2 KB out":                      8,
    "kernel TCP tx/rx for 4 packets":            12,
}
cpu_us  = sum(CPU_US.values())        # 79 us of CPU
wall_ms = 0.25 * 3 + 0.3              # 3 Redis round trips plus everything else

RPS, TARGET_UTIL = 1_000_000, 0.60
print(f"\nCPU/request {cpu_us} us   |   wall clock {wall_ms:.2f} ms")
print(f"cores at 100% util:      {RPS * cpu_us / 1e6:>9,.0f}")
print(f"cores at {TARGET_UTIL:.0%} util:       {RPS * cpu_us / 1e6 / TARGET_UTIL:>9,.0f}")
print(f"in flight (Little's Law):{RPS * wall_ms / 1000:>9,.0f}")
# CPU/request 79 us   |   wall clock 1.05 ms
# cores at 100% util:            79
# cores at 60% util:            132
# in flight (Little's Law):    1,050

# Now delete one line. Swap the ES256 verify for a cached HMAC at 2 us:
CPU_US["JWT verify (ES256, per request)"] = 2
cheaper = sum(CPU_US.values())        # 56 us
print(f"after the JWT change: {cheaper} us -> "
      f"{RPS * cheaper / 1e6 / TARGET_UTIL:,.0f} cores "
      f"({(132 - RPS * cheaper / 1e6 / TARGET_UTIL):,.0f} fewer)")
# after the JWT change: 56 us -> 93 cores (39 fewer)

Try it yourself

1. Run the script, then change the JWT line from 25 to 2 microseconds yourself and read the new core count. Multiply the difference by your instance's hourly price to see what one caching decision is worth per month, in both rupees and dollars.
2. Measure your real figure instead of trusting the table: perf stat -p $(pgrep -n your-service) -e cycles -- sleep 30, then divide the cycle count by the requests your metrics report for that window, and divide again by your clock in GHz to land on microseconds of CPU per request.
3. Put your measured CPU-per-request and your measured p50 wall clock side by side and take the ratio. That is your I/O-wait factor. Anything above 10 means the service is mostly waiting, and adding cores to it will change nothing.
4. Take the main-memory row at roughly 100 ns and work out how many cache misses fit inside your CPU budget. At 79 microseconds that is about 790 — so a single pass over a 1 MB structure with poor locality consumes most of the request on its own.
5. Check the clock you divided by under actual load with grep MHz /proc/cpuinfo or turbostat. If sustained all-core frequency is 20 percent below the boost number, your cycle budget shrank by 20 percent and every figure above moves with it.

References

• Latency Numbers Every Programmer Should Know (interactive, year-adjusted)
• perf-stat(1)
• Brendan Gregg — perf examples
• Brendan Gregg — the USE method