Dashboard/System design/19. Rate Limiter

SYSTEM DESIGN · LESSON 19 OF 52 · 11 MIN VIDEO · 4 TOPICS

Rate Limiter

Module Progress (0 / 4 Lessons Completed)0%
System design Track (0 / 52 Modules) · Overall 0/80 (0%)0%
⌁

Narrated HD Video Lesson (11 min) · Pro

Lesson Overview + Section #01 + RESHADED Checklist + Quiz #1 are unlocked free below (no login required). Unlock all 80 HD video lessons and deep-dive sections — or watch our two free sample video lessons first.

0% READSec 01/04 · System Design: The Rate Limiter
SYSTEM DESIGN VIDEO LESSON & GUIDE

Rate Limiter

A distributed rate limiter sits at the API edge to enforce per-client request quotas, preventing buggy retry storms, DDoS floods, and noisy neighbors from exhausting downstream compute and database capacity. At 1,000,000 checks/s across 10 million active clients, counter memory is surprisingly small (~400 MB), making sub-millisecond atomic concurrency, cluster-wide consistency, and failure posture (fail-open vs. fail-closed) the primary architectural drivers. L4–L6 interviewers test whether candidates can eliminate distributed read-modify-write race conditions using atomic Redis Lua scripts, navigate multi-region latency trade-offs, and match the five canonical counting algorithms to burst and memory constraints.

WHAT YOU WILL MASTER IN THIS LESSON
  • Specify HTTP 429 Too Many Requests edge contracts, composite throttling keys (user_id, IP, api_key, endpoint), and fail-open vs. fail-closed resilience policies.
  • Size a distributed rate-limiter state store (400 MB RAM vs. 10 Redis shards at 1,000,000 checks/s) to show why QPS rather than memory dictates sharding.
  • Design a lock-free distributed gateway architecture separating locally cached rule configs from atomic shared Redis counters (INCR and EVALSHA Lua scripts).
  • Implement and compare Token Bucket, Leaking Bucket, Fixed Window Counter, Sliding Window Log, and Sliding Window Counter across burst tolerance, edge accuracy, and per-client memory.
SYSTEMATIC BUILDING-BLOCK CHECKLIST (FROM MODULE 01)

4-Point Infrastructure Primitive Audit

Check off each dimension as you study this building block—interviewers test what it solves, how it works inside, what it costs, and how it fails.

0 / 4 Checked
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?

5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)

Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL

Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.

✗ NO-HIRE RED FLAG

Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.

2. Right-Sized Design
✓ STRONG HIRE SIGNAL

Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).

✗ NO-HIRE RED FLAG

Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.

3. Technical Depth
✓ STRONG HIRE SIGNAL

Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.

✗ NO-HIRE RED FLAG

Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.

4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL

Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'

✗ NO-HIRE RED FLAG

Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.

5. Communication
✓ STRONG HIRE SIGNAL

Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.

✗ NO-HIRE RED FLAG

Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.

ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 2Section 01: System Design: The Rate Limiter
ARCHITECTURE / DATA FLOW DIAGRAM
timed outClient Atimed outClient B10,000 per secondBuggy scriptall threads busyAPI serversDatabase
SECTION 01 OF 4

1. System Design: The Rate Limiter

★ FREE PREVIEW
Key Takeaway: An inline server-side rate limiter caps each client's request rate and sheds excess load with HTTP 429 before runaway scripts or attacks can starve shared application threads and databases.
SECTION 01 DIAGRAMS · SLIDE 1 OF 2System Design: The Rate Limiter
ARCHITECTURE / DATA FLOW DIAGRAM
timed outClient Atimed outClient B10,000 per secondBuggy scriptall threads busyAPI serversDatabase

The Noisy-Neighbor Outage & Edge Load Shedding

Consider a shared public API where each incoming call occupies an application worker thread and triggers a database query. Under normal traffic, every tenant receives sub-50 ms responses. If a single customer deploys a client update containing an unbounded retry loop that fires 10,000 RPS, server thread pools and database connections saturate within seconds, causing legitimate requests from thousands of other tenants to queue and time out—a classic noisy-neighbor outage with zero malicious intent. Placing a rate limiter in front of the API servers enforces a strict per-client cap (e.g., 100 requests/min): once the runaway client hits 100 within 10 ms, every subsequent request is rejected at the edge with HTTP 429 Too Many Requests and a Retry-After header before consuming backend CPU or IOPS.

  • Flood & abuse defense: Blocks volumetric Layer-7 DDoS floods, credential-stuffing attacks (5 login attempts/min), and accidental client retry storms.
  • Database & cost protection: Bounds peak downstream query load and caps expensive third-party or GPU inference calls per billing tier.
  • Multi-tenant fairness: Guarantees one heavy enterprise integration cannot degrade SLA tail latency for the rest of the customer base.

Choosing Rate-Limiting Key Dimensions

Production gateways rarely apply a single global counter; instead, rules evaluate composite keys combining caller identity and target resource sensitivity so strict security limits on authentication coexist with generous read quotas on search.

Throttling KeyExample RuleStrengthsEdge Cases & Pitfalls
Authenticated user_id100 writes / min / userAccurate per-account fairness across devices and networksRequires auth token parsing first; unauthenticated routes excluded
Client IP Address5 POST /login / min / IPProtects unauthenticated endpoints before DB/auth lookupCorporate NATs, university Wi-Fi, and mobile carrier CGNAT share 1 IP across thousands of users
API Key / Tenant Tier1,000 RPS (Enterprise) vs. 50 RPS (Free)Aligns directly with SaaS contractual billing tiersCompromised key requires instant revocation propagation
Composite (user + endpoint)POST /checkout: 10/min; GET /catalog: 600/minIsolates expensive mutations from cheap readsMultiplies active counter keys in memory by active endpoint count
Interview Signal: When throttling unauthenticated endpoints like /login by IP, pair a tight per-(IP, username) limit (5/min) with a looser IP-wide ceiling (100/min) so one employee typing a wrong password inside a corporate office NAT does not lock out 500 coworkers.
SECTION 02 OF 4

2. Requirements of a Rate Limiter's Design

🔒 PRO SECTION
Key Takeaway: Because a rate limiter runs inline on every API call (1,000,000 checks/s), it must add ~1 ms latency, share counters across all gateway instances (10 Redis shards for 400 MB of state), and explicitly choose between failing open or failing closed.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 03 OF 4

3. Design of a Rate Limiter

🔒 PRO SECTION
Key Takeaway: Separating background-synced rule caches from an atomic shared Redis counter store—updated via single-round-trip Lua scripts—prevents distributed race conditions without locking.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 04 OF 4

4. Rate Limiter Algorithms

🔒 PRO SECTION
Key Takeaway: Token Bucket and Sliding Window Counter dominate production API gateways because both require only two numbers (O(1) memory) per client while avoiding Fixed Window's 2x boundary burst and Sliding Log's O(limit) memory cost.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

ACTIVE RECALL & TRADE-OFF PROBES

System Design Checkpoint Quizzes (1)

CHECKPOINT QUIZ #1

In "System Design: The Rate Limiter" (Choosing Rate-Limiting Key Dimensions), how does a production-grade design handle "Client IP Address" (Edge Cases & Pitfalls)?