# AI & LLM System Design Interview Guide: ChatGPT Streaming, KV-Cache Routing & Enterprise RAG

> How to pass the modern GenAI architecture round at OpenAI, Anthropic, Meta, and Google: TTFT vs. TPOT budgets, continuous batching, hybrid vector search, and guardrails.

- **Author:** Former Amazon Software Engineer (Crush the FAANG Interview)
- **Category:** GenAI & LLM Systems
- **Updated:** 2026-11-20
- **Canonical URL:** https://crushthefaanginterview.com/blog/ai-llm-system-design-interview-guide-chatgpt-rag

## Key Takeaways
- Separate LLM latency into Time-To-First-Token (TTFT < 250ms, compute-bound prefill) and Time-Per-Output-Token (TPOT < 30ms/token, memory-bandwidth-bound decode) streamed over SSE.
- Prefix-aware L7 routing directs multi-turn chat turns to the GPU replica holding warm PagedAttention KV-cache blocks, slashing prefill FLOPs by up to 80%.
- Production RAG requires hybrid retrieval (BM25 lexical + dense HNSW vector search) followed by cross-encoder reranking and document-level ACL filtering before generation.

## 1. Why LLM Serving Differs from Traditional Stateless Microservices

A standard REST API handles a request in 15 milliseconds and frees its worker thread. An LLM completion holds GPU HBM memory for 5–25 seconds while generating hundreds of autoregressive tokens. Using standard round-robin load balancing destroys prefix KV-cache locality and causes head-of-line blocking.

| Bottleneck | Naive Design Failure | Production GenAI Pattern |
| --- | --- | --- |
| 5–20s full response wait | Buffered JSON HTTP response feels broken | Server-Sent Events (SSE) token streaming with < 250ms TTFT |
| GPU idle bubbles on variable sequence lengths | Static batching waits for longest prompt | Iteration-level continuous batching + PagedAttention (vLLM / TensorRT-LLM) |
| Re-computing 16K-token conversation history every turn | Random replica routing misses KV cache | Consistent prefix-hash routing + disaggregated prefill/decode pools |
| Hallucinated or unauthorized RAG citations | Pure cosine vector search ignores exact IDs & permissions | ACL pre-filter + BM25 + Dense Vector RRF fusion + Reranker + Citation grounding |

## 2. Back-of-the-Envelope GPU KV-Cache & Token Throughput Math

Interviewers love when you size GPU memory from first principles instead of waving at 'auto-scaling GPUs'. For a 70B-parameter FP16 model (~140 GB weights split across 2x H100 80GB GPUs via tensor parallelism), each concurrent 8,192-token context consumes ~2.5 GB of KV cache without GQA or ~320 MB with 8x Grouped-Query Attention (GQA). That immediately bounds max batch concurrency per node and justifies FP8 KV-cache quantization plus token-bucket rate limiting (TPM + RPM) at the API gateway.

## Frequently Asked Questions

### Do SWE system design interviews at FAANG test model training or inference systems?

General SWE and Infra interviews focus on production inference and application systems: SSE streaming, token-aware rate limiting, KV-cache routing, async ingestion pipelines, hybrid RAG retrieval, evaluation, and cost guardrails.

## Related Video & Interactive Courses
- [AI Building Blocks for System Design](https://crushthefaanginterview.com/course/sd-25-ai-building-blocks-for-system-design)
- [Design a ChatGPT System](https://crushthefaanginterview.com/course/sd-45-design-a-chatgpt-system)
- [LLM Customer Support Bot](https://crushthefaanginterview.com/course/sd-47-llm-powered-customer-support-bot-system-design)
- [AI-Powered Code Assistant](https://crushthefaanginterview.com/course/sd-48-ai-powered-code-assistant-system-design)
