AI & LLM System Design Interview Guide: ChatGPT Streaming, KV-Cache Routing & Enterprise RAG
How to pass the modern GenAI architecture round at OpenAI, Anthropic, Meta, and Google: TTFT vs. TPOT budgets, continuous batching, hybrid vector search, and guardrails.
- Separate LLM latency into Time-To-First-Token (TTFT < 250ms, compute-bound prefill) and Time-Per-Output-Token (TPOT < 30ms/token, memory-bandwidth-bound decode) streamed over SSE.
- Prefix-aware L7 routing directs multi-turn chat turns to the GPU replica holding warm PagedAttention KV-cache blocks, slashing prefill FLOPs by up to 80%.
- Production RAG requires hybrid retrieval (BM25 lexical + dense HNSW vector search) followed by cross-encoder reranking and document-level ACL filtering before generation.
1. Why LLM Serving Differs from Traditional Stateless Microservices
A standard REST API handles a request in 15 milliseconds and frees its worker thread. An LLM completion holds GPU HBM memory for 5β25 seconds while generating hundreds of autoregressive tokens. Using standard round-robin load balancing destroys prefix KV-cache locality and causes head-of-line blocking.
| Bottleneck | Naive Design Failure | Production GenAI Pattern |
|---|---|---|
| 5β20s full response wait | Buffered JSON HTTP response feels broken | Server-Sent Events (SSE) token streaming with < 250ms TTFT |
| GPU idle bubbles on variable sequence lengths | Static batching waits for longest prompt | Iteration-level continuous batching + PagedAttention (vLLM / TensorRT-LLM) |
| Re-computing 16K-token conversation history every turn | Random replica routing misses KV cache | Consistent prefix-hash routing + disaggregated prefill/decode pools |
| Hallucinated or unauthorized RAG citations | Pure cosine vector search ignores exact IDs & permissions | ACL pre-filter + BM25 + Dense Vector RRF fusion + Reranker + Citation grounding |
2. Back-of-the-Envelope GPU KV-Cache & Token Throughput Math
Interviewers love when you size GPU memory from first principles instead of waving at 'auto-scaling GPUs'. For a 70B-parameter FP16 model (~140 GB weights split across 2x H100 80GB GPUs via tensor parallelism), each concurrent 8,192-token context consumes ~2.5 GB of KV cache without GQA or ~320 MB with 8x Grouped-Query Attention (GQA). That immediately bounds max batch concurrency per node and justifies FP8 KV-cache quantization plus token-bucket rate limiting (TPM + RPM) at the API gateway.
Frequently Asked Questions
General SWE and Infra interviews focus on production inference and application systems: SSE streaming, token-aware rate limiting, KV-cache routing, async ingestion pipelines, hybrid RAG retrieval, evaluation, and cost guardrails.
Related Video Courses & Python 3.14 Workspaces
Drill every problem and architecture diagram from this guide inside Crush the FAANG Interview: