Pick a depth. Each prompt opens in your AI pre-loaded with the lesson. Click a row to preview the prompt.
Two pressures genuinely bite a chat system, and it is worth being deliberate about which ones do not. Duplicate requests are handled at the door by the client message ID from challenge 2, so duplicates are a solved problem rather than an open one. Sub-100ms reads are a design goal the whole system already meets, not a stress case. Data residency matters — chat carries some of the most personal data any system holds — but it is a partitioning decision made once at channel-creation time rather than something that changes under load. What actually breaks chat is traffic that concentrates, and topology that vanishes. Traffic concentrates in a way feeds never see: an incident starts, and forty thousand people converge on the same channel within ninety seconds, so the load is not merely a hundred times larger, it is a hundred times larger inside a single ordering domain owned by a single node. And topology vanishes in a way that is uniquely awkward for chat, because the ordering guarantee from challenge 3 requires exactly one owner per channel, and that owner lives somewhere specific. When somewhere specific goes away, the system has to answer a question it has been avoiding: is it acceptable for a channel to briefly refuse messages, or is it acceptable for two messages to share a sequence number? Those are the only two options, and pretending otherwise is how split-brain incidents get written up.
PRESSURE 1 — 100x, concentrated in one channel.
Scale. A workspace running at 580 sends/sec across thousands of channels. An outage starts. The #incident channel goes from roughly one message a minute to 20 a second, and its connected membership climbs from 400 to 30,000 as people pile in. Message volume across the workspace is up perhaps 4x. Deliveries in that one channel are up by a factor of about 90,000. And presence and typing indicators, which run at five to ten times message volume, spike alongside.
Bottleneck. Not storage: 20 appends a second is nothing. Not the sequencer: 20 allocations a second against a leased block is nothing. It is gateway CPU, and specifically serialisation — if the frame is encoded per recipient, 30,000 recipients times 20 messages a second is 600,000 encodes a second, which is the entire fleet. Behind that, socket write buffers back up, health check responses go slow, and the load balancer starts evicting healthy nodes for being slow rather than broken.
Decision. Three things, in this order. Encode once per message and share the byte slice — the single biggest win, and free. Coalesce per socket over a 50ms window for channels above a size threshold. Shed presence and typing to a 10-second aggregate, and drop them entirely above a load threshold, because they are lossy by nature and nobody files a bug about a missing typing indicator during an outage.
Tradeoff. Everyone in the busy channel pays up to 50ms of added delivery latency, and typing indicators become unreliable exactly when the channel is busiest. Against a p99 budget of 400ms, spending 50 to keep the other 350 is the right trade. It would not be if the budget were 120ms, which is why the window is per-channel and zero for DMs.
Incident. A real shape, and the important part is the second-order failure. Coalescing was configured but the threshold was set on channel membership rather than on connected sessions, so a channel with 50,000 members and 400 usually-connected sessions was already above it, while the incident channel crossed the connected threshold without crossing the membership one. Gateways hit CPU saturation on encoding. Health checks timed out. The load balancer evicted four of twelve gateway nodes. Thirty thousand clients reconnected within two seconds — and reconnect is far more expensive than steady state, because every client issues a resume across every channel it holds. The session registry, the channel-head reads and the catch-up range reads all took a simultaneous 30,000x spike. The remaining eight nodes saturated, and the eviction cascaded.
Outcome. Reconnect jitter of 0 to 30 seconds, so a reconnect storm becomes a reconnect drizzle. A token bucket on catch-up reads per gateway, with clients above the limit told to resync rather than streamed. Health checks moved to a dedicated thread that is not behind the encoding work, so a busy node reports busy rather than dead. And the coalescing threshold rewritten in terms of connected sessions.
PRESSURE 2 — the region that owned the numbering goes away.
Scale. Channel 991 is owned by a sequencer in ap-south-1. That region becomes unreachable — a network partition, not necessarily a failure, which is the awkward case.
Bottleneck. The ordering guarantee is “one owner per channel”. There is exactly one correct thing to do and two tempting wrong ones. Letting another region start assigning numbers gives two messages with seq 41221, which corrupts every cursor every client holds, permanently and undetectably. Waiting forever means the channel is dead until a human intervenes.
Decision. Ownership is a lease with a monotonic fencing token, held in a consensus store that is itself replicated across regions. Failover requires the lease to expire — a bounded window, 15 seconds, during which that channel refuses sends and returns a retryable error. Reads and history stay fully available throughout, because they touch no sequencer. The new owner resumes from the durably reserved high-water mark plus a gap reservation, and every write carries its fence: the store rejects any reservation whose fence is older than the one it has recorded. That last clause is what makes a partitioned old owner harmless rather than catastrophic — it may believe it is still in charge, and every one of its writes will be refused.
Tradeoff. Up to 15 seconds of “message not sending, retrying” on affected channels, versus permanently duplicated sequence numbers. A chat product can survive the first; it cannot survive the second, because the corruption is silent and unbounded. The honest alternative is per-region ID space with partial ordering, which is what federated systems like Matrix choose with an event DAG — a legitimate design, but it changes the product promise from “everyone sees the same order” to “everyone sees a compatible order”, and users notice.
Incident. A partition where the old owner could still reach the message store but not the lease service. Without fencing, both regions assigned from 41221. Six seconds of duplicate sequence numbers took four hours to reconcile, and every client cursor pointing into that range had to be invalidated. With fencing, the same partition is a 15-second send outage on the affected channels and no data problem at all.
Outcome. Failover budget of 15 seconds per channel; sends return a retryable error with a client-side automatic retry, so most users see a slow tick rather than an error. Channel ownership rebalances gradually rather than in one thundering wave, since moving a million channels at once is its own outage.
WHAT DOES NOT CHANGE, under either pressure. The durability rule: ack only after the durable write, in every region, under every load. The client’s idempotency key, which is why retries during a failover are safe. The storage schema and its bucketing. The read cursors, which is why history stays available while sends are refused. And the meaning of a sequence number — failover changes who assigns numbers, never what a number means.
// PRESSURE 2 — leased channel ownership with a fencing token.
// The fence, not the lease, is what makes split brain impossible: an
// old owner that is partitioned but alive will have every write refused.
const (
leaseTTL = 15 * time.Second
renewEvery = 5 * time.Second
safetyMargin = 3 * time.Second // refuse to assign inside this window
)
var ErrChannelUnavailable = errors.New("channel failing over; retry")
type Lease struct {
Channel string
Owner string
Fence uint64 // monotonic, incremented by the consensus store on grant
ExpiresAt time.Time
}
type Owner struct {
mu sync.RWMutex
self string
leases map[string]Lease
alloc *seq.Allocator // challenge 3
clock func() time.Time
}
func (o *Owner) Assign(ctx context.Context, channel string) (int64, error) {
o.mu.RLock()
l, held := o.leases[channel]
o.mu.RUnlock()
// Refuse INSIDE the safety margin, not at expiry. A number assigned
// at T-100ms may still be in flight to the store at T+400ms, by which
// point another region may legitimately own the channel.
if !held || o.clock().Add(safetyMargin).After(l.ExpiresAt) {
metrics.SequencerRefusals.WithLabelValues(channel).Inc()
return 0, ErrChannelUnavailable // retryable; the client retries with
} // the SAME client_msg_id, so a retry
// that lands on the new owner is safe
// The allocator carries the fence. The store rejects any reservation
// whose fence is older than the one it has recorded for this channel,
// so even a clock-skewed old owner cannot mint duplicates.
s, err := o.alloc.Next(ctx, channel)
if errors.Is(err, seq.ErrLostOwnership) {
o.mu.Lock()
delete(o.leases, channel) // we were fenced; stop pretending
o.mu.Unlock()
return 0, ErrChannelUnavailable
}
return s, err
}
// Handover on the new owner. Resume from the DURABLE high-water mark,
// not from the last number anyone remembers issuing, then reserve a gap
// so any write still in flight from the old owner cannot collide.
func (o *Owner) Acquire(ctx context.Context, channel string, fence uint64) error {
head, err := o.store.DurableHighWater(ctx, channel)
if err != nil {
return err
}
const inFlightGap = 1000 // holes are free (challenge 3); collisions are not
if err := o.store.SetHighWater(ctx, channel, head+inFlightGap, fence); err != nil {
return err
}
o.mu.Lock()
o.leases[channel] = Lease{channel, o.self, fence, o.clock().Add(leaseTTL)}
o.mu.Unlock()
return nil
}go run main.go