SYSTEM DESIGN · LESSON 18 OF 52 · 11 MIN VIDEO · 3 TOPICS

Pub-Sub

Module Progress (0 / 3 Lessons Completed)0%
System design Track (0 / 52 Modules) · Overall 0/80 (0%)0%
⌁

Narrated HD Video Lesson (11 min) · Pro

Lesson Overview + Section #01 + RESHADED Checklist + Quiz #1 are unlocked free below (no login required). Unlock all 80 HD video lessons and deep-dive sections — or watch our two free sample video lessons first.

0% READSec 01/03 · System Design: The Pub-Sub Abstraction
SYSTEM DESIGN VIDEO LESSON & GUIDE

Pub-Sub

Publish-Subscribe (Pub-Sub) decouples event producers from an arbitrary number of independent downstream systems—write once to a named topic, and fan out to every subscribed reader without coupling services in space or time. At scale, high fan-out creates extreme read amplification (50:1 egress-to-ingress ratios), making naive per-subscriber queue copying prohibitively expensive in storage and IOPS. L4–L6 interviewers look for candidates who replace message duplication with an immutable, partitioned append-only log (Apache Kafka / Google Cloud Pub/Sub style) where each subscriber tracks progress via a lightweight 64-bit offset while partition leaders and followers guarantee durability and causal entity ordering.

WHAT YOU WILL MASTER IN THIS LESSON
  • Distinguish point-to-point competing-consumer queues from topic-based fan-out pub-sub across space decoupling, time decoupling, and historical replay.
  • Quantify fan-out write-vs-read asymmetry (10 MB/s ingress turning into 500 MB/s egress across 50 subscribers) and explain why zero-copy single-log storage is required.
  • Design a partitioned append-only commit log using deterministic key hashing (hash(entity_id) % P) to preserve per-entity causal order while scaling horizontally.
  • Architect a fault-tolerant broker cluster coordinating Partition Leaders, In-Sync Replicas (ISR), a consensus Cluster Manager (ZooKeeper/KRaft), and an Offset Consumer Manager.
SYSTEMATIC BUILDING-BLOCK CHECKLIST (FROM MODULE 01)

4-Point Infrastructure Primitive Audit

Check off each dimension as you study this building block—interviewers test what it solves, how it works inside, what it costs, and how it fails.

0 / 4 Checked
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?

5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)

Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL

Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.

✗ NO-HIRE RED FLAG

Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.

2. Right-Sized Design
✓ STRONG HIRE SIGNAL

Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).

✗ NO-HIRE RED FLAG

Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.

3. Technical Depth
✓ STRONG HIRE SIGNAL

Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.

✗ NO-HIRE RED FLAG

Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.

4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL

Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'

✗ NO-HIRE RED FLAG

Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.

5. Communication
✓ STRONG HIRE SIGNAL

Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.

✗ NO-HIRE RED FLAG

Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.

ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 2Section 01: System Design: The Pub-Sub Abstraction
ARCHITECTURE / DATA FLOW DIAGRAM
knows everyone?Ride servicepublisherQueueone consumer per messagegot itBillingmissedReceiptsmissedAnalytics
SECTION 01 OF 3

1. System Design: The Pub-Sub Abstraction

★ FREE PREVIEW
Key Takeaway: Whereas a point-to-point queue hands each message to a single worker, a pub-sub topic broadcasts every published event independently to every subscribed system while decoupling publishers and subscribers in both space and time.
SECTION 01 DIAGRAMS · SLIDE 1 OF 2System Design: The Pub-Sub Abstraction
ARCHITECTURE / DATA FLOW DIAGRAM
knows everyone?Ride servicepublisherQueueone consumer per messagegot itBillingmissedReceiptsmissedAnalytics

Why Point-to-Point Queues Fail at Multi-Service Broadcast

Consider a ride-hailing platform where completing a trip (ride_ended) must trigger Billing, Receipts, and Analytics. Publishing ride_ended into a single point-to-point messaging queue fails immediately: Billing dequeues and deletes the message, starving Receipts and Analytics. Having Ride Service push three separate RPCs or enqueue into three hardcoded queues violates service boundaries—every time a new team (such as Fraud Check) needs ride_ended events, engineers must modify, test, and redeploy Ride Service. Pub-Sub introduces a named Topic intermediary: Ride Service publishes once to ride_ended, and Billing, Receipts, Analytics, and Fraud Check subscribe independently.

DimensionPoint-to-Point Messaging QueuePublish-Subscribe (Pub-Sub) Topic
Delivery Semantics1 message → 1 competing consumer1 message → N independent subscribers (fan-out)
Adding a New ConsumerSteals share of traffic from existing workersReceives full independent event stream; zero publisher code changes
Post-Read LifecycleDeleted immediately upon consumer ACKRetained until retention window expires (enables replay)
Canonical Use CaseTask distribution across identical worker poolsDomain event broadcast, CDC, telemetry & stream pipelines

Decoupling in Space and Time

Topic-based routing provides two orthogonal forms of architectural isolation that keep large microservice graphs resilient during deployments and outages.

  • Decoupling in space: Publishers know only the topic_id—never how many subscribers exist, where they run, or whether new subscribers (Fraud Check) attach dynamically at runtime.
  • Decoupling in time: Because the topic durably retains events on disk, if Analytics goes offline for 4 hours during a schema migration, Ride Service and Billing continue unaffected; once Analytics restarts, it drains the missed 4-hour backlog at its own pace.

Delivery Mechanics: Broker Push vs. Subscriber Pull

Pub-Sub engines deliver messages either by having brokers push events over webhooks/streams to subscribers or by letting subscribers pull batches from brokers. Push minimizes idle latency for low-volume notifications, whereas pull dominates high-throughput log brokers because slow or recovering subscribers control their own consumption rate without needing complex broker-side backpressure.

Delivery ModeEnd-to-End LatencyFlow Control & BackpressureBatching & Catch-Up Efficiency
Push DeliveryLowest (immediate broker dispatch)Hard: fast broker can overwhelm slow subscribers without rate windowsModerate; requires broker retry queues for offline targets
Pull Delivery (Log)Low with long-polling (~1–10 ms)Natural: each subscriber fetches only when capacity is freeHigh: lagging readers pull large sequential MB batches from disk
Interview Signal: Even in pull-based systems like Kafka, long-polling (max_wait_ms = 100 ms, min_bytes = 1) eliminates tight empty-poll CPU spin while preserving consumer-controlled flow control.
SECTION 02 OF 3

2. Introduction to Pub-Sub

🔒 PRO SECTION
Key Takeaway: Because fan-out multiplies outbound traffic by the subscriber count—turning 10 MB/s of writes into 500 MB/s of reads across 50 subscribers—a scalable pub-sub engine must store each message once and optimize for sequential multi-reader throughput.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 03 OF 3

3. Design of a Pub-Sub System

🔒 PRO SECTION
Key Takeaway: Splitting each topic into replicated append-only partition logs and tracking a single 64-bit integer offset per subscriber per partition eliminates N-way storage duplication while guaranteeing causal ordering per partition key.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

ACTIVE RECALL & TRADE-OFF PROBES

System Design Checkpoint Quizzes (1)

CHECKPOINT QUIZ #1

In "System Design: The Pub-Sub Abstraction" (Why Point-to-Point Queues Fail at Multi-Service Broadcast), how does a production-grade design handle "Adding a New Consumer" (Publish-Subscribe (Pub-Sub) Topic)?