Dashboard/System design/45. Design a ChatGPT System

SYSTEM DESIGN · LESSON 45 OF 52 · 15 MIN VIDEO · 2 TOPICS

Design a ChatGPT System

Module Progress (0 / 2 Lessons Completed)0%
System design Track (0 / 52 Modules) · Overall 0/80 (0%)0%
⌁

Narrated HD Video Lesson (15 min) · Pro

Lesson Overview + Section #01 + RESHADED Checklist + Quiz #1 are unlocked free below (no login required). Unlock all 80 HD video lessons and deep-dive sections — or watch our two free sample video lessons first.

0% READSec 01/02 · System Design: ChatGPT System
SYSTEM DESIGN VIDEO LESSON & GUIDE

Design a ChatGPT System

Designing a conversational LLM platform like ChatGPT (100M DAU, 1B prompts/day, 23,000 RPS peak, 5.8M output tokens/s) inverts traditional web architecture: database storage (2.4 TB/day) and stateless CPU services are inexpensive, while autoregressive GPU inference across 1,160-2,300 8x80 GB GPU nodes dominates both tail latency and unit economics. Because generating a 500-token (~375-word) reply takes ~10 seconds sequentially rather than milliseconds, every design choice serves two core laws: stream every token immediately over HTTP Server-Sent Events (TTFT < 1 s), and keep every GPU slot saturated via iteration-level continuous batching and strict KV-cache memory budgets. L4-L6 interviewers evaluate your mastery of prefill vs. decode mechanics, HBM KV-cache math, continuous batching, summarized context windows, and mid-stream GPU crash failover.

WHAT YOU WILL MASTER IN THIS LESSON
  • Size GPU server fleets (1,160 average / 2,300 peak 8-GPU nodes) and concurrent open streams (116,000) from token generation rates (5.8M tokens/s) and HBM KV-cache capacity (~380 concurrent 4K-token chats per 640 GB node).
  • Isolate expensive GPU workers behind token-based API rate limiters, stateless chat orchestrators, Cassandra conversation partitions, and fast pre-inference safety classifiers.
  • Distinguish compute-bound parallel Prefill (TTFT) from memory-bandwidth-bound sequential Decode, raising GPU slot utilization from ~56% (static batching) toward ~100% with Continuous Batching.
  • Enforce fixed prompt token budgets via backward sliding windows plus small-model summarization, and recover transparently from mid-stream GPU crashes using prompt-prefix prefill replay and SSE Last-Event-ID buffers.
SYSTEMATIC INTERVIEW CHECKLIST · RESHADED + 45-MIN CLOCK

8-Step RESHADED Execution Checklist

Follow this exact sequence from the Intro narration on every system design prompt so you never miss requirements, math, contracts, or the 13-minute deep dive.

0 / 8 Checked
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?

5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)

Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL

Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.

✗ NO-HIRE RED FLAG

Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.

2. Right-Sized Design
✓ STRONG HIRE SIGNAL

Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).

✗ NO-HIRE RED FLAG

Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.

3. Technical Depth
✓ STRONG HIRE SIGNAL

Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.

✗ NO-HIRE RED FLAG

Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.

4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL

Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'

✗ NO-HIRE RED FLAG

Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.

5. Communication
✓ STRONG HIRE SIGNAL

Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.

✗ NO-HIRE RED FLAG

Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.

ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 1Section 01: System Design: ChatGPT System
ARCHITECTURE / DATA FLOW DIAGRAM
≈ 375 wordsUserbrowser or appsafety checksChat appservers≈ 10 secondsLanguage modelon GPUs
SECTION 01 OF 2

1. System Design: ChatGPT System

★ FREE PREVIEW
Key Takeaway: Autoregressive LLM generation takes ~10 seconds per 500-token reply (5.8M tokens/s fleet-wide across 1,160-2,300 8-GPU servers), making low-latency token streaming (TTFT < 1 s) and GPU HBM/batch efficiency the overriding architectural drivers.
ARCHITECTURE / DATA FLOW DIAGRAM
≈ 375 wordsUserbrowser or appsafety checksChat appservers≈ 10 secondsLanguage modelon GPUs

Why Autoregressive LLM Serving Breaks Request-Response Web Design

Unlike a standard microservice that fetches rows from a database and returns JSON in 10 ms, a Large Language Model (LLM) generates text autoregressively one token (~0.75 words) at a time on specialized GPUs. Each generated token requires tens of milliseconds of matrix-vector computation across tens of billions of parameters, so a typical 500-token (~375-word) explanation takes ~10 seconds end-to-end. Blocking silently for 10 seconds feels broken to users, so responses must stream token by token the instant each is decoded. At the same time, stateless model calls require re-sending multi-turn chat history with each prompt, inputs and streamed outputs must pass safety classifiers, and scarce GPU time makes every architectural decision directly a unit-cost decision.

  • Token Granularity: 1 token ≈ 3/4 word; all rate limits, context windows, memory budgets, and billing units are measured in tokens rather than HTTP request counts.
  • Perceived vs. Total Latency: Users judge responsiveness by Time to First Token (TTFT < 1 s) and smooth Inter-Token Latency (~50 tokens/s, faster than human reading speed), not total 10 s completion time.
  • Latency vs. Cost Tension: Packing larger batches onto each GPU lowers $/token dramatically, but pushing batch size too far stretches per-step latency and degrades streaming smoothness.

Functional and Non-Functional Requirements

A complete conversational AI product combines stateless token generation with stateful multi-turn conversation history, safety guardrails, and user feedback collection (thumbs up/down and regenerate) that feeds downstream evaluation and post-training pipelines. At 1,000+ GPU nodes, individual GPU HBM errors and node crashes occur daily, requiring mid-stream fault recovery without dropping active user sessions.

CategoryRequirementTarget & System Implication
FunctionalPrompt & Streaming ReplyAccept natural-language prompts and stream tokens incrementally as generated.
FunctionalMulti-Turn ContextInclude prior conversation turns so follow-up pronouns and instructions resolve naturally.
FunctionalPersistent Chat HistoryList past conversations paginated newest-first, reopen any thread, and continue seamlessly.
FunctionalFeedback & RegenerationRecord per-message ratings or trigger alternate completions for RLHF and quality evals.
Non-FunctionalLow Streaming LatencyTTFT < 1 s (p95) and sustained >= 30-50 tokens/s decode throughput.
Non-FunctionalHigh AvailabilitySurvive daily GPU worker crashes and brief mobile network drops mid-stream.
Non-FunctionalSafety & PrivacyFilter harmful prompts pre-GPU and harmful outputs in-stream; isolate tenant conversations.
Non-FunctionalGPU Cost EfficiencyMaximize useful tokens/s per GPU via continuous batching, KV budgets, and model tiering.

Back-of-the-Envelope Capacity and GPU Fleet Estimation

Assuming 100,000,000 (100M) DAU sending 10 prompts/day yields 1,000,000,000 (1B) requests/day—averaging 1,000,000,000 / 86,400 s ≈ 11,600 RPS and doubling during peak daytime hours to ~23,000 RPS. Generating an average 500-token reply demands 11,600 * 500 = 5,800,000 (5.8M) output tokens/s on average (11.6M tokens/s at peak). If one 8-GPU server (e.g., 8x80 GB H100/A100 running continuous batching) sustains ~5,000 output tokens/s, baseline inference requires 5,800,000 / 5,000 = 1,160 GPU servers (9,280 GPUs) and ~2,300 GPU servers (18,400 GPUs) at peak. Because each 500-token answer streams at 50 tokens/s for 10 s, Little's Law dictates 11,600 RPS * 10 s = 116,000 open streaming connections on average (230,000 peak), while storing ~600 tokens/exchange * 4 B = 2.4 KB across 1B daily turns adds a modest 2.4 TB/day (876 TB/year).

Resource DimensionFormulaAverage LoadPeak / Annual Scale
Prompt Request Rate100M DAU * 10 / 86,400 s11,600 RPS23,000 RPS (2x peak)
Output Token Throughput11,600 RPS * 500 tokens5.8M tokens/s11.6M tokens/s peak
8-GPU Inference Servers5.8M tok/s / 5,000 tok/s/node1,160 servers (9,280 GPUs)2,300 servers (18,400 GPUs)
Concurrent Open Streams11,600 RPS * (500 / 50 tok/s)116,000 open streams~230,000 open streams peak
Conversation Text Storage1B/day * 600 tok * 4 B2.4 TB/day876 TB/year
Interview Signal: Rate-Limiting Rule: Enforce user quotas at the API Gateway in tokens per minute/day, never raw HTTP request counts—one 8,000-token code-review prompt consumes 100x the prefill FLOPs and KV-cache HBM of an 80-token question.
SECTION 02 OF 2

2. Design of a ChatGPT System

🔒 PRO SECTION
Key Takeaway: Shield expensive GPU workers behind stateless chat orchestrators, Cassandra conversation partitions, fast moderation models, and an iteration-level continuous-batching scheduler that bounds KV-cache memory via summarized context budgets and replays partial prefixes on worker crashes.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

ACTIVE RECALL & TRADE-OFF PROBES

System Design Checkpoint Quizzes (1)

CHECKPOINT QUIZ #1

In "System Design: ChatGPT System" (Functional and Non-Functional Requirements), how does a production-grade design handle "Functional" (Target & System Implication)?