Chat Requirements: The Numbers That Shape the Design

Why this matters

Chat is the system people most often assume they could build in a weekend, and it is one of the few where that assumption is spectacularly wrong. Open a socket, relay a string — done. What turns it into a system is the set of requirements nobody writes on the whiteboard. The message has to arrive in the same order for everyone, including the person who scrolls up three years later. It has to survive the sender’s phone dying between the tap and the acknowledgement, without either losing the message or posting it twice. It has to land on four devices belonging to one human, three of which are asleep. The unread badge has to be exactly right, because a wrong badge is the single most reported bug in every messaging product ever shipped — people forgive a slow app and never forgive a red dot that lies. And the latency budget is unforgiving in a way that batch systems are not: an email notification can take thirty seconds and nobody notices, but chat that takes eight hundred milliseconds feels broken, because a human is watching a typing indicator and has a mental model of how fast the other person types. Every design choice in this module is downstream of those five sentences, so it is worth counting before you draw.

Demo

Take a single large workspace and put numbers on it before naming a single component. The number that matters is not the send rate — it is the delivery rate, which is sends multiplied by how many recipients are connected right now. That multiplier is the entire design problem, and it is invisible until you compute it.

Run the model below and watch the two rates diverge. Sends are a few hundred per second, which any database on earth will absorb without complaint. Deliveries are tens of thousands per second, which is a fleet decision. Storage, meanwhile, turns out to be the cheapest part of the whole system — a fact that surprises people who expect the message table to be the scary bit.

Also note the cost line. Chat is a connection-heavy system, and connections cost money whether anyone is talking or not: an idle socket still occupies memory and a file descriptor for eight hours of a working day.

# Capacity model for one large workspace. Change the inputs, not the code.
DAU                  = 500_000    # daily active people
PEAK_CONNECTED_FRAC  = 0.60       # fraction with a live socket at peak
MSGS_PER_ACTIVE_DAY  = 20         # sends per person per day
PEAK_MULTIPLIER      = 5.0        # peak second vs daily average second
AVG_CHANNEL_MEMBERS  = 40         # members of a typical channel receiving a send
MEMBER_CONNECTED_FRAC= 0.40       # of those members, how many are online
AVG_MSG_BYTES        = 400        # body + metadata, stored
REPLICAS             = 3
BYTES_PER_SOCKET     = 30_000     # goroutine/event-loop state + read/write buffers
SOCKETS_PER_GATEWAY  = 50_000

connected   = DAU * PEAK_CONNECTED_FRAC
sends_day   = DAU * MSGS_PER_ACTIVE_DAY
sends_avg   = sends_day / 86_400
sends_peak  = sends_avg * PEAK_MULTIPLIER

# THE number: one send becomes many socket writes.
fanout      = AVG_CHANNEL_MEMBERS * MEMBER_CONNECTED_FRAC
deliv_peak  = sends_peak * fanout

store_day_gb = sends_day * AVG_MSG_BYTES / 1e9
store_yr_tb  = store_day_gb * 365 * REPLICAS / 1000

gw_mem_gb   = connected * BYTES_PER_SOCKET / 1e9
gateways    = -(-connected // SOCKETS_PER_GATEWAY)      # ceil
gateways_ha = gateways * 2                              # N+N across two zones

# Dual currency. A 4 vCPU / 8 GB node, on-demand, is roughly:
USD_NODE_MONTH = 70
INR_PER_USD    = 66.5

print(f"peak connected sockets   {connected:>12,.0f}")
print(f"sends/sec  avg / peak    {sends_avg:>12,.0f} / {sends_peak:,.0f}")
print(f"fanout multiplier        {fanout:>12,.1f} x")
print(f"DELIVERIES/sec at peak   {deliv_peak:>12,.0f}")
print(f"storage/day              {store_day_gb:>12,.1f} GB")
print(f"storage/year (x{REPLICAS})     {store_yr_tb:>12,.1f} TB")
print(f"socket memory, fleet     {gw_mem_gb:>12,.1f} GB")
print(f"gateway nodes (HA)       {gateways_ha:>12,.0f}")
cost = gateways_ha * USD_NODE_MONTH
print(f"gateway fleet / month    ${cost:,.0f}  ~ INR {cost*INR_PER_USD:,.0f}")

# peak connected sockets            300,000
# sends/sec  avg / peak                 116 / 579
# fanout multiplier                    16.0 x
# DELIVERIES/sec at peak              9,259
# storage/day                           4.0 GB
# storage/year (x3)                     4.4 TB
# socket memory, fleet                  9.0 GB
# gateway nodes (HA)                       12
# gateway fleet / month                  $840  ~ INR 55,860

Try it yourself

1. Set AVG_CHANNEL_MEMBERS to 5000 and MEMBER_CONNECTED_FRAC to 0.5 and note the deliveries/sec figure — that single change is the whole of challenge 5.
2. Raise PEAK_CONNECTED_FRAC to 0.95 (a company-wide all-hands) and record how many gateway nodes the model now demands, and what that costs in both USD and INR.
3. Set AVG_MSG_BYTES to 4000 to model rich text plus reactions plus link previews, and check whether yearly storage has become expensive relative to the gateway fleet cost.
4. Compute sends/sec and deliveries/sec for a two-person DM (AVG_CHANNEL_MEMBERS = 2, MEMBER_CONNECTED_FRAC = 0.9) and write down the ratio between that and the 5000-member case.
5. Find the value of MEMBER_CONNECTED_FRAC at which deliveries/sec crosses 50,000, and decide from the latency budget whether coalescing or more gateway nodes is the correct response.

References

• Slack Engineering — Real Time Messaging
• Discord — How Discord Scaled Elixir to 5,000,000 Concurrent Users
• RFC 6455 — The WebSocket Protocol