Crush the FAANG Interview
See Plans β†’
GENAI & LLM SYSTEMS11 min read Β· Updated 2026-11-20 Β· View Raw Markdown (.md)

AI & LLM System Design Interview Guide: ChatGPT Streaming, KV-Cache Routing & Enterprise RAG

How to pass the modern GenAI architecture round at OpenAI, Anthropic, Meta, and Google: TTFT vs. TPOT budgets, continuous batching, hybrid vector search, and guardrails.

KEY ENGINEERING TAKEAWAYS
  • Separate LLM latency into Time-To-First-Token (TTFT < 250ms, compute-bound prefill) and Time-Per-Output-Token (TPOT < 30ms/token, memory-bandwidth-bound decode) streamed over SSE.
  • Prefix-aware L7 routing directs multi-turn chat turns to the GPU replica holding warm PagedAttention KV-cache blocks, slashing prefill FLOPs by up to 80%.
  • Production RAG requires hybrid retrieval (BM25 lexical + dense HNSW vector search) followed by cross-encoder reranking and document-level ACL filtering before generation.

1. Why LLM Serving Differs from Traditional Stateless Microservices

A standard REST API handles a request in 15 milliseconds and frees its worker thread. An LLM completion holds GPU HBM memory for 5–25 seconds while generating hundreds of autoregressive tokens. Using standard round-robin load balancing destroys prefix KV-cache locality and causes head-of-line blocking.

BottleneckNaive Design FailureProduction GenAI Pattern
5–20s full response waitBuffered JSON HTTP response feels brokenServer-Sent Events (SSE) token streaming with < 250ms TTFT
GPU idle bubbles on variable sequence lengthsStatic batching waits for longest promptIteration-level continuous batching + PagedAttention (vLLM / TensorRT-LLM)
Re-computing 16K-token conversation history every turnRandom replica routing misses KV cacheConsistent prefix-hash routing + disaggregated prefill/decode pools
Hallucinated or unauthorized RAG citationsPure cosine vector search ignores exact IDs & permissionsACL pre-filter + BM25 + Dense Vector RRF fusion + Reranker + Citation grounding

2. Back-of-the-Envelope GPU KV-Cache & Token Throughput Math

Interviewers love when you size GPU memory from first principles instead of waving at 'auto-scaling GPUs'. For a 70B-parameter FP16 model (~140 GB weights split across 2x H100 80GB GPUs via tensor parallelism), each concurrent 8,192-token context consumes ~2.5 GB of KV cache without GQA or ~320 MB with 8x Grouped-Query Attention (GQA). That immediately bounds max batch concurrency per node and justifies FP8 KV-cache quantization plus token-bucket rate limiting (TPM + RPM) at the API gateway.

INTERVIEW FOLLOW-UP FAQ

Frequently Asked Questions

Do SWE system design interviews at FAANG test model training or inference systems?

General SWE and Infra interviews focus on production inference and application systems: SSE streaming, token-aware rate limiting, KV-cache routing, async ingestion pipelines, hybrid RAG retrieval, evaluation, and cost guardrails.

CONTINUE PRACTICING IN THE PLATFORM

Related Video Courses & Python 3.14 Workspaces

Drill every problem and architecture diagram from this guide inside Crush the FAANG Interview: