How Load Balancers Pick a Server

Why this matters

Round-robin sounds fair until one backend's GC pause makes it the slowest in the pool — round-robin happily keeps sending it 1/N of traffic anyway. Picking the wrong algorithm means your p99 latency is set by your slowest replica even when 9 out of 10 are healthy.

Demo

Configure and inspect different LB algorithms on a managed cloud load balancer.

Try it yourself

1. Switch an ALB target group from round-robin to least-outstanding and run a load test where one backend sleeps 200ms — observe traffic redistribute.
2. Set up weighted routing 95/5 between two target groups and confirm with 1000 curls that the split lands within 1% of expected.
3. Toggle SourceIP distribution and verify the same client always hits the same backend across 50 sequential requests.

References

• AWS — ALB routing algorithms
• Envoy — Load balancing types