Dashboard/System design/08. Load Balancers

SYSTEM DESIGN · LESSON 8 OF 52 · 5 MIN VIDEO · 3 TOPICS

Load Balancers

Module Progress (0 / 3 Lessons Completed)0%
System design Track (0 / 52 Modules) · Overall 0/80 (0%)0%
⌁

Narrated HD Video Lesson (5 min) · Pro

Lesson Overview + Section #01 + RESHADED Checklist + Quiz #1 are unlocked free below (no login required). Unlock all 80 HD video lessons and deep-dive sections — or watch our two free sample video lessons first.

0% READSec 01/03 · Introduction to Load Balancers
SYSTEM DESIGN VIDEO LESSON & GUIDE

Load Balancers

Load balancers are the traffic-management backbone of horizontal scaling, presenting a single externally reachable Virtual IP (VIP) while distributing millions of concurrent connections across fleets of ephemeral backend servers and geographically dispersed data centers. Beyond uniform traffic spreading, modern load-balancing tiers enforce sub-second health-check isolation, terminate TLS encryption, and prevent cascading overload during regional outages. In L4-L6 interviews, candidates must distinguish global (GSLB) from local (SLB) routing, select the right OSI layer (L4 vs. L7) and scheduling algorithm for stateful vs. stateless protocols, and eliminate the load balancer itself as a single point of failure.

WHAT YOU WILL MASTER IN THIS LESSON
  • Size backend server pools with N + k failover headroom and place load balancers across edge, application, and data-replica tiers
  • Evaluate global routing strategies—DNS Round-Robin, health-aware GeoDNS, and BGP Anycast—alongside local data-center VIP forwarding
  • Choose between static and dynamic routing algorithms, stateful and stateless session affinity, and Layer 4 transport vs. Layer 7 application proxies in a multi-tier ingress stack
SYSTEMATIC BUILDING-BLOCK CHECKLIST (FROM MODULE 01)

4-Point Infrastructure Primitive Audit

Check off each dimension as you study this building block—interviewers test what it solves, how it works inside, what it costs, and how it fails.

0 / 4 Checked
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?

5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)

Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL

Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.

✗ NO-HIRE RED FLAG

Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.

2. Right-Sized Design
✓ STRONG HIRE SIGNAL

Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).

✗ NO-HIRE RED FLAG

Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.

3. Technical Depth
✓ STRONG HIRE SIGNAL

Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.

✗ NO-HIRE RED FLAG

Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.

4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL

Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'

✗ NO-HIRE RED FLAG

Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.

5. Communication
✓ STRONG HIRE SIGNAL

Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.

✗ NO-HIRE RED FLAG

Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.

ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 2Section 01: Introduction to Load Balancers
ARCHITECTURE / DATA FLOW DIAGRAM
which one?ClientsServermax 5,000 req/sServerServerLoad balancer
SECTION 01 OF 3

1. Introduction to Load Balancers

★ FREE PREVIEW
Key Takeaway: A load balancer abstracts a dynamic pool of horizontally scaled servers behind one stable Virtual IP, continuously ejecting unhealthy nodes and offloading cross-cutting concerns like TLS termination so capacity grows and heals invisibly to clients.
SECTION 01 DIAGRAMS · SLIDE 1 OF 2Introduction to Load Balancers
ARCHITECTURE / DATA FLOW DIAGRAM
which one?ClientsServermax 5,000 req/sServerServerLoad balancer

Capacity Math and Why Clients Cannot Self-Balance

Suppose an e-commerce flash sale drives 50,000 RPS, while CPU and memory profiling show a single application instance saturates at 5,000 RPS before p99 latency spikes. Basic division dictates 50,000 RPS / 5,000 RPS = 10 servers just to carry peak load; provisioning N + 2 or 25% headroom (12-13 servers) ensures that losing two machines simultaneously does not push survivors past 100% utilization into a cascading crash. Exposing all 13 server IPs directly to mobile and browser clients fails in production: clients lack real-time visibility into per-server CPU load or crashes, leaking internal topology widens the attack surface, and DNS caches prevent rapid removal of dead hosts. Placing a Load Balancer in front exposes one stable Virtual IP (VIP) and routes every incoming request exclusively to responsive instances.

  • Active health checks: The balancer polls an application endpoint (GET /healthz every 2 s) and removes an instance after 2-3 consecutive timeouts (4-6 s detection window).
  • Passive health checks (outlier detection): The balancer monitors live responses and immediately ejects any server returning spikes of HTTP 502/503 errors or TCP resets without waiting for the next timer tick.
  • Graceful connection draining: During rolling deployments, instances are marked draining so new requests route elsewhere while in-flight transactions finish cleanly before shutdown.

Multi-Tier Placement Across the Architecture

Load balancing is not restricted to the internet-facing perimeter (North-South traffic); independent load-balancing layers decouple every horizontally scaled tier inside a data center (East-West traffic). An external edge balancer distributes public client connections across web/API gateway nodes, an internal balancer spreads RPC calls from gateways across stateless business-logic microservices, and a data-tier balancer (or connection proxy like PgBouncer / ProxySQL) distributes read queries across asynchronous database replicas. Isolating each tier behind its own internal VIP allows engineering teams to autoscale compute-heavy services (50 instances) independently from lightweight gateway fleets (8 instances).

# Example upstream pool definition with active probes & TLS offload
upstream checkout_app_pool {
    least_conn;
    server 10.0.1.11:8080 max_fails=3 fail_timeout=6s;
    server 10.0.1.12:8080 max_fails=3 fail_timeout=6s;
    server 10.0.1.13:8080 max_fails=3 fail_timeout=6s;
}
# Edge VIP 203.0.113.10:443 terminates TLS 1.3 -> proxies HTTP/2 to 10.0.1.x

Core Guarantees and Perimeter Offloading

In addition to horizontal scalability, fault masking, and latency smoothing, edge load balancers centralize computationally expensive and security-critical network tasks. Terminating TLS 1.3 handshakes and managing X.509 certificate rotation at the balancer relieves backend application runtimes from asymmetric cryptographic overhead and allows inspection or re-encryption over persistent mTLS connection pools inside the private VPC.

Load Balancer FunctionUnderlying MechanismProduction Engineering Benefit
Horizontal ScalabilityDynamic backend registration via service discoveryAdd or remove servers during traffic spikes with zero client config changes
High AvailabilityActive probes + passive 5xx outlier ejectionMasks single-host hardware crashes within seconds
TLS TerminationHardware/AES-NI accelerated TLS handshake & cert renewalCentralizes certificate management and frees backend CPU cores
L3/L4 Flood MitigationSYN cookies, connection rate caps, malformed packet dropsAbsorbs volumetric transport floods before sockets exhaust app servers
Golden-Signal TelemetryPer-route RPS, p50/p95/p99 latency, and status histogramsDrives horizontal autoscalers and SLO burn-rate alerts uniformly
Interview Signal: Never leave a single load balancer box on an architecture diagram in an interview: always specify at least an Active-Passive pair sharing a floating VIP via VRRP/heartbeats, or an Active-Active cluster fronted by BGP ECMP routers.
SECTION 02 OF 3

2. Global and Local Load Balancing

🔒 PRO SECTION
Key Takeaway: Multi-region architectures split routing into two stages: Global Server Load Balancing (GSLB) steers users to the nearest healthy data center via health-aware GeoDNS or Anycast, while Local Load Balancing (SLB) distributes packets across private backend servers inside that region.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 03 OF 3

3. Advanced Details of Load Balancers

🔒 PRO SECTION
Key Takeaway: Production load-balancing stacks layer stateless L4 transport balancers in front of content-aware L7 application proxies, selecting dynamic algorithms like Least Connections or EWMA latency when request costs vary across servers.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

ACTIVE RECALL & TRADE-OFF PROBES

System Design Checkpoint Quizzes (1)

CHECKPOINT QUIZ #1

In "Introduction to Load Balancers" (Core Guarantees and Perimeter Offloading), how does a production-grade design handle "High Availability" (Production Engineering Benefit)?