Lesson Overview + Section #01 + RESHADED Checklist + Quiz #1 are unlocked free below (no login required). Unlock all 80 HD video lessons and deep-dive sections — or watch our two free sample video lessons first.
An enterprise LLM customer support system must resolve millions of daily inquiries in natural language without hallucinating company policies, leaking cross-tenant PII, or executing unauthorized financial actions. At 2M conversations/day (10M messages/day, 350 RPS peak, and 32B tokens/day), naive calls to a frontier LLM incur $38,000/day in inference spend and risk prompt-injected refund abuse. L4–L6 interviewers evaluate how candidates ground generation via hybrid Retrieval-Augmented Generation (RAG), enforce deterministic authorization boundaries around tool execution, meet < 1 s time-to-first-token (TTFT), and route traffic across tiered models with seamless human-agent escalation.
WHAT YOU WILL MASTER IN THIS LESSON
Quantify message throughput (350 RPS peak), daily token budgets (30B input / 2B output), vector index memory (4 GB), and tiered model routing economics ($38K/day -> $14.1K/day).
Isolate LLM reasoning from state mutation using customer-scoped OAuth credentials, deterministic code-level policy checks, user confirmation gates, and idempotency keys.
Enforce bidirectional safety via PII/injection input guards, streaming sentence-level groundedness verification, multi-provider fallbacks, and context-preserving human handoff.
Follow this exact sequence from the Intro narration on every system design prompt so you never miss requirements, math, contracts, or the 13-minute deep dive.
0 / 8 Checked
RRequirements & ScopeMin 2–8
Clarify who the users are, 3–4 core functional flows, explicit non-goals, and target latency/availability SLAs before drawing a single box.
Avoid trap: Still gathering requirements at minute 15 squeezes the deep dive where L5/L6 offers are decided.
EBack-of-the-Envelope MathMin 8–12
Compute ONLY numbers that actively change architectural choices: DAU → peak read/write RPS, daily & 5-year storage, bandwidth (or tokens/sec & KV-cache VRAM for AI).
Avoid trap: Spending 10 minutes multiplying irrelevant byte counts that never influence sharding or caching.
SStorage & Data ModelMin 12–15
Define core tables, primary/partition keys, and access indexes. Separate small structured metadata rows (~1 KB) from bulk binary blobs (S3 + CDN).
Avoid trap: Picking Cassandra or NoSQL by buzzword when 40 writes/s & 3 TB fit cleanly on one indexed SQL primary + replicas.
AAPIs & Wire ContractsMin 15–17
Write 3–4 concrete REST/gRPC/SSE/WebSocket endpoints with request/response shapes, cursor pagination, and Idempotency-Key headers on mutations.
Avoid trap: Hand-waving client-server contracts and discovering missing parameters halfway through the deep dive.
HHigh-Level DesignMin 17–27
Draw the simplest end-to-end working read and write paths first. Every box on the board must trace back to a stated requirement or number.
Avoid trap: Reciting a memorized 12-microservice Kafka mesh before proving why a simpler synchronous path breaks.
DDetailed Design (Deep Dive)Min 27–36
Proactively audit your high-level diagram, name the top 2 bottlenecks (hot partitions, fan-out skew, lock contention, GPU saturation), and go deep on internals.
Avoid trap: Waiting passively for the interviewer to rescue pacing or skimming every box at shallow surface depth.
EEvaluation & Failure ModesMin 36–41
State what each architectural choice COSTS you, and trace graceful degradation when cache, queue, external vendor, or primary DB fails.
Avoid trap: Assuming happy-path uptime or placing slow 3-second external calls inline on revenue-critical write paths.
DDistinctive FeatureMin 41–45
Solve the signature hard problem unique to this prompt (e.g., double-entry ledger idempotency, Geohash dispatch locks, OT/CRDT sync, or SSE token streaming).
Avoid trap: Spending 20 minutes on generic login/CRUD tables and zero time on the core distributed challenge.
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?
5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)
Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL
Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.
✗ NO-HIRE RED FLAG
Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.
2. Right-Sized Design
✓ STRONG HIRE SIGNAL
Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).
✗ NO-HIRE RED FLAG
Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.
3. Technical Depth
✓ STRONG HIRE SIGNAL
Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.
✗ NO-HIRE RED FLAG
Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.
4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL
Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'
✗ NO-HIRE RED FLAG
Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.
5. Communication
✓ STRONG HIRE SIGNAL
Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.
✗ NO-HIRE RED FLAG
Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.
ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 1Section 01: System Design: LLM-Powered Customer Support Bot
ARCHITECTURE / DATA FLOW DIAGRAM
SECTION 01 OF 2
1. System Design: LLM-Powered Customer Support Bot
★ FREE PREVIEW
Key Takeaway: Treat the language model strictly as a natural-language reasoning and synthesis engine while surrounding infrastructure supplies verified knowledge, authenticated business actions, cost routing, and human fallback.
ARCHITECTURE / DATA FLOW DIAGRAM
Problem Framing: Why Human-Only, Decision Trees, and Raw LLMs Fail
Consider a global e-commerce platform handling 2M support conversations per day covering parcel tracking, returns, and billing disputes. Scaling human support desks linearly is prohibitively expensive ($5–$12 per ticket) and leaves customers queued during flash sales or outages, while rigid decision-tree chatbots fail on nuanced phrasing with frustrating fallback loops. A standalone Large Language Model (LLM) parses free-form intent naturally, yet possesses zero knowledge of proprietary store policies or live order tables; left ungrounded, it confidently fabricates refund windows (hallucination), creating legal and financial liability. Production architecture bridges this gap by retrieving authoritative help-center passages, querying live order APIs under customer permissions, and escalating unresolved threads to human agents with full conversation context.
Support Paradigm
Natural Language Flexibility
Marginal Cost / Chat
Primary Failure Mode
Human Support Desk
High (handles ambiguity & empathy)
$5.00–$12.00
Cannot burst 3x instantly during outages; high operational spend
Rule-Based Decision Tree
Very Low (rigid button/keyword paths)
< $0.001
Fails on multi-intent or off-script queries ("I didn't understand")
Standalone Raw LLM
High (fluent prose)
~$0.02
Hallucinates store policies; blind to live order state; vulnerable to injection
Grounded RAG + Tool-Calling Bot
High (fluent + cited + actionable)
~$0.007–$0.019
Requires retrieval latency budgeting, policy guardrails, and human fallback
Functional & Non-Functional Requirements
Customers expect a coherent multi-turn conversation across web widgets, mobile apps, and email that both answers policy questions with citations and executes state-changing workflows (address updates, returns, refunds). On the non-functional side, factual groundedness supersedes fluency: abstaining or escalating is strictly preferable to guessing. Interactive chat additionally requires streaming the first response token within <= 1 s (TTFT), surviving 3x traffic surges during site incidents, and preventing cross-customer data leakage or prompt-injected privilege escalation.
Functional — Grounded Q&A: Answer policy and product questions strictly from current company documentation with inline source citations.
Functional — Authenticated Actions: Execute read/write workflows (get_order, update_address, start_return, issue_refund) against live backend APIs.
Functional — Multi-Turn & Omnichannel State: Resolve coreferences across turns ("what about the other item?") seamlessly across web, mobile, and email.
Functional — Context-Rich Human Handoff: Escalate smoothly on user request, low confidence, or high-risk disputes with an automated structured summary.
Non-Functional — Factual Groundedness: Zero unverified policy claims; enforce explicit abstention when retrieval confidence falls below threshold.
Non-Functional — Streaming Latency: Achieve p95 TTFT <= 1.0 s via token streaming so conversations feel instantaneous.
Non-Functional — Availability & Safety: Degrade gracefully during LLM provider outages, mask PII, block prompt injections, and cap token spend.
Back-of-the-Envelope Scale, Storage, and Token Cost Estimation
At 2M conversations/day and 5 user messages per conversation, the system processes 10M messages/day, equating to 10M / 86,400 s ≈ 116 RPS average and ~350 RPS at 3x peak sale load. Assuming each turn sends 3,000 input tokens (system rules, recent turns, retrieved passages) and generates 200 output tokens, daily token volume hits 30B input tokens and 2B output tokens. Pricing a frontier model at $1 / 1M input tokens and $4 / 1M output tokens yields (30,000 * $1) + (2,000 * $4) = $30,000 + $8,000 = $38,000/day (~$0.019 per chat)—orders of magnitude cheaper than human agents, yet dominant enough in infrastructure spend to justify model routing. By contrast, indexing 50,000 help articles split into 20 chunks each produces 1M chunks; at 1,024 float32 dimensions (4 KB/vector), the entire vector index occupies just 4 GB, fitting effortlessly in RAM on a single replicated node.
Interview Signal: Highlighting that the entire 1M-chunk vector index fits in 4 GB of RAM immediately shows L5/L6 maturity: you avoid over-engineering petabyte vector sharding and focus deep-dive time on the true bottlenecks—LLM TTFT latency, hallucination control, and $38K/day inference cost.
SECTION 02 OF 2
2. Design of an LLM-Powered Customer Support Bot
🔒 PRO SECTION
Key Takeaway: Center the runtime on a stateless orchestrator backed by hybrid vector+keyword RAG, deterministic policy-checked tool execution, bidirectional guardrails, and tiered 70/30 model routing that meets a 710 ms TTFT budget at $14.1K/day.
Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.
In "System Design: LLM-Powered Customer Support Bot" (Problem Framing: Why Human-Only, Decision Trees, and Raw LLMs Fail), how does a production-grade design handle "Rule-Based Decision Tree" (Primary Failure Mode)?