Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
A million requests a second is a phrase people repeat without ever converting it into anything physical, and the conversion is the whole skill. Two numbers come first, and almost everyone conflates them. The first is supply: a core delivers exactly one core-second of work per second, so a 64-core box has 64 core-seconds a second to spend, and spreading that over a million requests leaves 64 microseconds of CPU each — about 192,000 cycles at 3 GHz, which sounds enormous until you learn that a single TLS handshake costs a million of them. The second is occupancy: Little's Law says the number of requests alive inside your system at any instant is the arrival rate multiplied by how long each one stays, so a million a second with a 20 ms mean response time means twenty thousand requests are in there right now, holding sockets, buffers and heap. Sizing cores off latency instead of CPU is a three-hundred-fold error, and it is made constantly — someone reads 20 ms, multiplies by a million, concludes they need twenty thousand cores, and either abandons the project or buys a fleet they will never use. Get these two conversions apart and every later number in this course falls out of them.
Work the two conversions on a single 64-core box. Notice that the first answer is measured in microseconds of CPU and the second in thousands of concurrent requests, and that changing the response time moves only one of them.
# budget.py — what a million requests a second costs you in time
RPS = 1_000_000
CORES = 64 # one modern dual-socket-class box, 64 physical cores
GHZ = 3.0 # sustained all-core clock, not the boost number on the box
# 1. CPU budget per request, if this ONE box served all of it.
# A core delivers exactly one core-second of work per second. That is the
# only supply you have.
core_seconds_per_second = CORES * 1.0
cpu_budget_s = core_seconds_per_second / RPS
print(f"CPU budget: {cpu_budget_s * 1e6:.1f} us/request "
f"({cpu_budget_s * GHZ * 1e9:,.0f} cycles)")
# CPU budget: 64.0 us/request (192,000 cycles)
# 2. Little's Law: L = lambda * W.
# W is WALL CLOCK time in the system — including every millisecond spent
# waiting on a socket. It is not CPU time. Mixing the two is the single
# most common capacity mistake.
for w_ms in (1, 20, 200):
L = RPS * (w_ms / 1000)
print(f"W={w_ms:>3} ms -> {L:>9,.0f} requests in flight")
# W= 1 ms -> 1,000 requests in flight
# W= 20 ms -> 20,000 requests in flight
# W=200 ms -> 200,000 requests in flight
# 3. What that in-flight state costs in memory.
PER_REQ_BYTES = 64 * 1024 # socket buffers + parser state + response buffer
L = RPS * 0.020
print(f"in-flight memory at W=20ms: {L * PER_REQ_BYTES / 1e9:.2f} GB")
# in-flight memory at W=20ms: 1.31 GB
# 4. The same arithmetic per box, once the load is spread.
BOXES = 20
print(f"per box: {RPS / BOXES:,.0f} rps, {RPS / BOXES * 0.020:,.0f} in flight")
# per box: 50,000 rps, 1,000 in flightCORES to 16 and re-run. The budget drops to 16 microseconds per request. Compare that against one main-memory miss at roughly 80 to 100 ns: you get about 200 of them, total, for the entire request.w_ms with your own service's measured p50 wall-clock latency, taken from your metrics rather than guessed. The in-flight count that comes out is what actually sizes your worker count and your connection pool — not your rps.ulimit -n and then ss -s on a box you own. Compare the total socket count against the in-flight number Little's Law predicts for that box's share of traffic. If sockets vastly exceed in-flight requests, you are holding idle keep-alive connections; count them, because they cost memory whether or not they carry a request.PER_REQ_BYTES honestly. cat /proc/sys/net/ipv4/tcp_rmem and cat /proc/sys/net/ipv4/tcp_wmem print the minimum, default and maximum socket buffer sizes in bytes. Add the two defaults plus your response size and see what the memory line becomes.