Lesson Overview + Section #01 + RESHADED Checklist + Quiz #1 are unlocked free below (no login required). Unlock all 80 HD video lessons and deep-dive sections — or watch our two free sample video lessons first.
Designing a conversational LLM platform like ChatGPT (100M DAU, 1B prompts/day, 23,000 RPS peak, 5.8M output tokens/s) inverts traditional web architecture: database storage (2.4 TB/day) and stateless CPU services are inexpensive, while autoregressive GPU inference across 1,160-2,3008x80 GB GPU nodes dominates both tail latency and unit economics. Because generating a 500-token (~375-word) reply takes ~10 seconds sequentially rather than milliseconds, every design choice serves two core laws: stream every token immediately over HTTP Server-Sent Events (TTFT < 1 s), and keep every GPU slot saturated via iteration-level continuous batching and strict KV-cache memory budgets. L4-L6 interviewers evaluate your mastery of prefill vs. decode mechanics, HBM KV-cache math, continuous batching, summarized context windows, and mid-stream GPU crash failover.
WHAT YOU WILL MASTER IN THIS LESSON
Size GPU server fleets (1,160 average / 2,300 peak 8-GPU nodes) and concurrent open streams (116,000) from token generation rates (5.8M tokens/s) and HBM KV-cache capacity (~380 concurrent 4K-token chats per 640 GB node).
Isolate expensive GPU workers behind token-based API rate limiters, stateless chat orchestrators, Cassandra conversation partitions, and fast pre-inference safety classifiers.
Distinguish compute-bound parallel Prefill (TTFT) from memory-bandwidth-bound sequential Decode, raising GPU slot utilization from ~56% (static batching) toward ~100% with Continuous Batching.
Enforce fixed prompt token budgets via backward sliding windows plus small-model summarization, and recover transparently from mid-stream GPU crashes using prompt-prefix prefill replay and SSE Last-Event-ID buffers.
Follow this exact sequence from the Intro narration on every system design prompt so you never miss requirements, math, contracts, or the 13-minute deep dive.
0 / 8 Checked
RRequirements & ScopeMin 2–8
Clarify who the users are, 3–4 core functional flows, explicit non-goals, and target latency/availability SLAs before drawing a single box.
Avoid trap: Still gathering requirements at minute 15 squeezes the deep dive where L5/L6 offers are decided.
EBack-of-the-Envelope MathMin 8–12
Compute ONLY numbers that actively change architectural choices: DAU → peak read/write RPS, daily & 5-year storage, bandwidth (or tokens/sec & KV-cache VRAM for AI).
Avoid trap: Spending 10 minutes multiplying irrelevant byte counts that never influence sharding or caching.
SStorage & Data ModelMin 12–15
Define core tables, primary/partition keys, and access indexes. Separate small structured metadata rows (~1 KB) from bulk binary blobs (S3 + CDN).
Avoid trap: Picking Cassandra or NoSQL by buzzword when 40 writes/s & 3 TB fit cleanly on one indexed SQL primary + replicas.
AAPIs & Wire ContractsMin 15–17
Write 3–4 concrete REST/gRPC/SSE/WebSocket endpoints with request/response shapes, cursor pagination, and Idempotency-Key headers on mutations.
Avoid trap: Hand-waving client-server contracts and discovering missing parameters halfway through the deep dive.
HHigh-Level DesignMin 17–27
Draw the simplest end-to-end working read and write paths first. Every box on the board must trace back to a stated requirement or number.
Avoid trap: Reciting a memorized 12-microservice Kafka mesh before proving why a simpler synchronous path breaks.
DDetailed Design (Deep Dive)Min 27–36
Proactively audit your high-level diagram, name the top 2 bottlenecks (hot partitions, fan-out skew, lock contention, GPU saturation), and go deep on internals.
Avoid trap: Waiting passively for the interviewer to rescue pacing or skimming every box at shallow surface depth.
EEvaluation & Failure ModesMin 36–41
State what each architectural choice COSTS you, and trace graceful degradation when cache, queue, external vendor, or primary DB fails.
Avoid trap: Assuming happy-path uptime or placing slow 3-second external calls inline on revenue-critical write paths.
DDistinctive FeatureMin 41–45
Solve the signature hard problem unique to this prompt (e.g., double-entry ledger idempotency, Geohash dispatch locks, OT/CRDT sync, or SSE token streaming).
Avoid trap: Spending 20 minutes on generic login/CRUD tables and zero time on the core distributed challenge.
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?
5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)
Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL
Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.
✗ NO-HIRE RED FLAG
Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.
2. Right-Sized Design
✓ STRONG HIRE SIGNAL
Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).
✗ NO-HIRE RED FLAG
Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.
3. Technical Depth
✓ STRONG HIRE SIGNAL
Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.
✗ NO-HIRE RED FLAG
Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.
4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL
Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'
✗ NO-HIRE RED FLAG
Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.
5. Communication
✓ STRONG HIRE SIGNAL
Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.
✗ NO-HIRE RED FLAG
Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.
ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 1Section 01: System Design: ChatGPT System
ARCHITECTURE / DATA FLOW DIAGRAM
SECTION 01 OF 2
1. System Design: ChatGPT System
★ FREE PREVIEW
Key Takeaway: Autoregressive LLM generation takes ~10 seconds per 500-token reply (5.8M tokens/s fleet-wide across 1,160-2,3008-GPU servers), making low-latency token streaming (TTFT < 1 s) and GPU HBM/batch efficiency the overriding architectural drivers.
ARCHITECTURE / DATA FLOW DIAGRAM
Why Autoregressive LLM Serving Breaks Request-Response Web Design
Unlike a standard microservice that fetches rows from a database and returns JSON in 10 ms, a Large Language Model (LLM) generates text autoregressively one token (~0.75 words) at a time on specialized GPUs. Each generated token requires tens of milliseconds of matrix-vector computation across tens of billions of parameters, so a typical 500-token (~375-word) explanation takes ~10 seconds end-to-end. Blocking silently for 10 seconds feels broken to users, so responses must stream token by token the instant each is decoded. At the same time, stateless model calls require re-sending multi-turn chat history with each prompt, inputs and streamed outputs must pass safety classifiers, and scarce GPU time makes every architectural decision directly a unit-cost decision.
Token Granularity: 1 token ≈ 3/4 word; all rate limits, context windows, memory budgets, and billing units are measured in tokens rather than HTTP request counts.
Perceived vs. Total Latency: Users judge responsiveness by Time to First Token (TTFT < 1 s) and smooth Inter-Token Latency (~50 tokens/s, faster than human reading speed), not total 10 s completion time.
Latency vs. Cost Tension: Packing larger batches onto each GPU lowers $/token dramatically, but pushing batch size too far stretches per-step latency and degrades streaming smoothness.
Functional and Non-Functional Requirements
A complete conversational AI product combines stateless token generation with stateful multi-turn conversation history, safety guardrails, and user feedback collection (thumbs up/down and regenerate) that feeds downstream evaluation and post-training pipelines. At 1,000+ GPU nodes, individual GPU HBM errors and node crashes occur daily, requiring mid-stream fault recovery without dropping active user sessions.
Category
Requirement
Target & System Implication
Functional
Prompt & Streaming Reply
Accept natural-language prompts and stream tokens incrementally as generated.
Functional
Multi-Turn Context
Include prior conversation turns so follow-up pronouns and instructions resolve naturally.
Functional
Persistent Chat History
List past conversations paginated newest-first, reopen any thread, and continue seamlessly.
Functional
Feedback & Regeneration
Record per-message ratings or trigger alternate completions for RLHF and quality evals.
Non-Functional
Low Streaming Latency
TTFT < 1 s (p95) and sustained >= 30-50 tokens/s decode throughput.
Non-Functional
High Availability
Survive daily GPU worker crashes and brief mobile network drops mid-stream.
Maximize useful tokens/s per GPU via continuous batching, KV budgets, and model tiering.
Back-of-the-Envelope Capacity and GPU Fleet Estimation
Assuming 100,000,000 (100M) DAU sending 10 prompts/day yields 1,000,000,000 (1B) requests/day—averaging 1,000,000,000 / 86,400 s ≈ 11,600 RPS and doubling during peak daytime hours to ~23,000 RPS. Generating an average 500-token reply demands 11,600 * 500 = 5,800,000 (5.8M) output tokens/s on average (11.6M tokens/s at peak). If one 8-GPU server (e.g., 8x80 GB H100/A100 running continuous batching) sustains ~5,000 output tokens/s, baseline inference requires 5,800,000 / 5,000 = 1,160 GPU servers (9,280 GPUs) and ~2,300 GPU servers (18,400 GPUs) at peak. Because each 500-token answer streams at 50 tokens/s for 10 s, Little's Law dictates 11,600 RPS * 10 s = 116,000 open streaming connections on average (230,000 peak), while storing ~600 tokens/exchange * 4 B = 2.4 KB across 1B daily turns adds a modest 2.4 TB/day (876 TB/year).
Resource Dimension
Formula
Average Load
Peak / Annual Scale
Prompt Request Rate
100M DAU * 10 / 86,400 s
11,600 RPS
23,000 RPS (2x peak)
Output Token Throughput
11,600 RPS * 500 tokens
5.8M tokens/s
11.6M tokens/s peak
8-GPU Inference Servers
5.8M tok/s / 5,000 tok/s/node
1,160 servers (9,280 GPUs)
2,300 servers (18,400 GPUs)
Concurrent Open Streams
11,600 RPS * (500 / 50 tok/s)
116,000 open streams
~230,000 open streams peak
Conversation Text Storage
1B/day * 600 tok * 4 B
2.4 TB/day
876 TB/year
Interview Signal: Rate-Limiting Rule: Enforce user quotas at the API Gateway in tokens per minute/day, never raw HTTP request counts—one 8,000-token code-review prompt consumes 100x the prefill FLOPs and KV-cache HBM of an 80-token question.
SECTION 02 OF 2
2. Design of a ChatGPT System
🔒 PRO SECTION
Key Takeaway: Shield expensive GPU workers behind stateless chat orchestrators, Cassandra conversation partitions, fast moderation models, and an iteration-level continuous-batching scheduler that bounds KV-cache memory via summarized context budgets and replays partial prefixes on worker crashes.
Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.
In "System Design: ChatGPT System" (Functional and Non-Functional Requirements), how does a production-grade design handle "Functional" (Target & System Implication)?