Dashboard/System design/16. Distributed Cache

SYSTEM DESIGN · LESSON 16 OF 52 · 11 MIN VIDEO · 6 TOPICS

Distributed Cache

Module Progress (0 / 6 Lessons Completed)0%
System design Track (0 / 52 Modules) · Overall 0/80 (0%)0%
⌁

Narrated HD Video Lesson (11 min) · Pro

Lesson Overview + Section #01 + RESHADED Checklist + Quiz #1 are unlocked free below (no login required). Unlock all 80 HD video lessons and deep-dive sections — or watch our two free sample video lessons first.

0% READSec 01/06 · System Design: The Distributed Cache
SYSTEM DESIGN VIDEO LESSON & GUIDE

Distributed Cache

A distributed cache stores hot working-set data across a cluster of commodity RAM servers to serve reads in under 1 ms while shielding primary databases from read saturation. At scale—millions of requests per second across terabytes of state—the core engineering challenges shift from simple key-value lookups to consistent-hash routing, write-consistency policies, bounded-staleness replication, and constant-time eviction. In L4–L6 interviews, candidates are evaluated on whether they explicitly define their write and eviction policies, prevent cache stampedes during shard topology changes, and justify trading linearizability for sub-millisecond availability.

WHAT YOU WILL MASTER IN THIS LESSON
  • Quantify read-latency reductions and database offload ratios using cache hit-rate math and dual-constraint (memory vs. QPS) fleet sizing.
  • Select appropriate write policies (write-through, write-back, write-around, cache-aside) and eviction strategies (LRU, LFU, TTL) based on workload consistency and durability tolerances.
  • Design a shard-routing and high-availability layer using consistent hashing with virtual nodes, primary-replica replication, and ZooKeeper/etcd membership management.
  • Implement single-node O(1) lookup and eviction internals combining a hash table, doubly linked list, lazy TTL checks, and probabilistic active sweeping.
  • Compare Memcached and Redis across data models, threading architectures, persistence, sharding, and failover to choose the right engine for a production workload.
SYSTEMATIC BUILDING-BLOCK CHECKLIST (FROM MODULE 01)

4-Point Infrastructure Primitive Audit

Check off each dimension as you study this building block—interviewers test what it solves, how it works inside, what it costs, and how it fails.

0 / 4 Checked
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?

5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)

Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL

Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.

✗ NO-HIRE RED FLAG

Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.

2. Right-Sized Design
✓ STRONG HIRE SIGNAL

Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).

✗ NO-HIRE RED FLAG

Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.

3. Technical Depth
✓ STRONG HIRE SIGNAL

Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.

✗ NO-HIRE RED FLAG

Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.

4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL

Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'

✗ NO-HIRE RED FLAG

Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.

5. Communication
✓ STRONG HIRE SIGNAL

Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.

✗ NO-HIRE RED FLAG

Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.

ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 1Section 01: System Design: The Distributed Cache
ARCHITECTURE / DATA FLOW DIAGRAM
UsersmissApp servermemory fullCachenode 1Cachenode 2Cachenode 310 ms per queryDatabase
SECTION 01 OF 6

1. System Design: The Distributed Cache

★ FREE PREVIEW
Key Takeaway: A distributed cache partitions hot data in memory across multiple servers so application reads resolve in under 1 ms while reducing database query load by an order of magnitude.
ARCHITECTURE / DATA FLOW DIAGRAM
UsersmissApp servermemory fullCachenode 1Cachenode 2Cachenode 310 ms per queryDatabase

The Database Read Bottleneck & Cache-Aside Mechanics

Consider an e-commerce catalog receiving 50,000 RPS on product detail pages. Routing every read directly to a disk-backed relational database incurs roughly 10 ms per query and rapidly exhausts connection pools, buffer caches, and disk IOPS. Placing an in-memory cache adjacent to the application tier exploits the latency hierarchy: raw DRAM access takes ~100 ns, and even with serialization and an intra-datacenter TCP round trip, a remote cache read completes in < 1 ms. Under the standard cache-aside pattern, the application checks the cache first; on a cache hit, the value returns immediately without touching the database, and on a cache miss, the application queries the database, populates the cache for subsequent readers, and returns the result.

  • Cache hit path: Application hashes the key, fetches the serialized payload from RAM over TCP in ~1 ms, and completely bypasses the database.
  • Cache miss path: Cache lookup returns MISS (1 ms), application queries the primary database (10 ms), writes the result back to the cache asynchronously or inline, and returns in ~11 ms.
  • Single-node limitation: A standalone cache server is bounded by single-machine DRAM (64–512 GB) and creates a catastrophic single point of failure—if it crashes, all 50,000 RPS hit the cold database simultaneously.
Interview Signal: Never size a primary database only for steady-state post-cache miss traffic without addressing cold-cache startup and node failure; an unmitigated burst from 5,000 RPS back to 50,000 RPS will trigger cascading database timeouts.

Quantifying Latency & Database Offload

Expected read latency in a cache-fronted architecture is governed by the weighted average E[L] = h * L_hit + (1 - h) * (L_hit + L_db), where h is the hit ratio. At 50,000 RPS with a 90% hit ratio (h = 0.90), L_hit = 1 ms, and L_db = 10 ms, average read latency drops from 10 ms to 0.90 * 1 ms + 0.10 * 11 ms = 2.0 ms—a 5x latency improvement. Simultaneously, database read throughput drops by 90%, from 50,000 RPS down to 50,000 * 0.10 = 5,000 RPS, allowing the persistence tier to be provisioned for write durability rather than repetitive read fan-out.

Traffic PathShare of ReadsLatency FormulaEffective LatencyDatabase Load (at 50,000 RPS)
Uncached Baseline100%L_db10.0 ms50,000 RPS
Cache Hit90%L_hit1.0 ms0 RPS
Cache Miss10%L_hit + L_db11.0 ms5,000 RPS
Blended Cached System100%0.9 * 1 ms + 0.1 * 11 ms2.0 ms (5x faster)5,000 RPS (90% offload)

Why Distribute Across Dedicated Cache Nodes?

Applications can embed an in-process local cache (such as Caffeine or Guava) inside each app server's heap, eliminating network hops entirely (< 1 us), or deploy a dedicated distributed cache cluster shared across all app servers. In-process caches duplicate hot keys across dozens of app instances, suffer low hit rates when load balancers spray requests randomly, and serve inconsistent state because invalidating a key across 100 stateless web servers requires broadcast coordination. Partitioning a shared cache across dedicated servers pools aggregate RAM into a multi-terabyte keyspace where every key has a deterministic home shard.

  • Memory pooling: Sharding keys across N cache nodes provides N * RAM_per_node of non-overlapping working-set capacity rather than repeating the same hot items in every app process.
  • Independent lifecycle: Deploying or autoscaling stateless application servers does not wipe warm cache state or trigger miss storms.
  • Tiered L1/L2 hybrid: Ultra-high-scale systems occasionally combine a tiny 100–500 ms TTL in-process L1 cache for extreme celebrity hot keys with a shared distributed L2 cache for working-set capacity.
SECTION 02 OF 6

2. Background of Distributed Cache

🔒 PRO SECTION
Key Takeaway: Every distributed cache architecture is defined by four foundational choices: how writes propagate between cache and database, which keys are evicted under memory pressure, when stale entries expire via TTL, and how consistent hashing places keys across shards.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 03 OF 6

3. High-Level Design of a Distributed Cache

🔒 PRO SECTION
Key Takeaway: Sizing a distributed cache requires evaluating both memory working-set capacity and peak request throughput independently, taking the stricter bound as the shard count before multiplying by the replication factor.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 04 OF 6

4. Detailed Design of a Distributed Cache

🔒 PRO SECTION
Key Takeaway: Production distributed caches pair ZooKeeper/etcd membership coordination and asynchronous primary-replica shards at the cluster layer with an O(1) hash-map plus doubly-linked-list LRU engine inside each node.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 05 OF 6

5. Evaluation of a Distributed Cache's Design

🔒 PRO SECTION
Key Takeaway: By trading strict linearizability for asynchronous 3-way replication and consistent hashing, the design achieves sub-millisecond reads, < 5% key movement during scale-out, and 99.9999% per-shard availability.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 06 OF 6

6. Memcached versus Redis

🔒 PRO SECTION
Key Takeaway: Choose Memcached for simple, vertically scalable, multithreaded caching of opaque binary blobs with minimal memory fragmentation; choose Redis when your workload requires rich server-side data structures, atomic Lua scripts, persistence, or built-in replication and failover.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

ACTIVE RECALL & TRADE-OFF PROBES

System Design Checkpoint Quizzes (1)

CHECKPOINT QUIZ #1

In "System Design: The Distributed Cache" (Quantifying Latency & Database Offload), how does a production-grade design handle "Cache Hit" (Database Load (at 50,000 RPS))?