Dashboard/System design/25. AI Building Blocks for System Design

SYSTEM DESIGN · LESSON 25 OF 52 · 11 MIN VIDEO · 7 TOPICS

AI Building Blocks for System Design

Module Progress (0 / 7 Lessons Completed)0%
System design Track (0 / 52 Modules) · Overall 0/80 (0%)0%
⌁

Narrated HD Video Lesson (11 min) · Pro

Lesson Overview + Section #01 + RESHADED Checklist + Quiz #1 are unlocked free below (no login required). Unlock all 80 HD video lessons and deep-dive sections — or watch our two free sample video lessons first.

0% READSec 01/07 · Models as Components in a System
SYSTEM DESIGN VIDEO LESSON & GUIDE

AI Building Blocks for System Design

Modern system design treats machine learning classifiers, multi-stage rankers, vector indexes, RAG pipelines, and generative LLMs as concrete distributed components governed by strict latency, token, memory, and reliability budgets. Unlike deterministic microservices, AI components execute probabilistic inference on specialized GPU hardware, require online feature parity, and introduce non-deterministic failure modes. In L4-L6 interviews, candidates are evaluated on isolating inference behind versioned services with deterministic fallbacks, multi-stage funnel math, RAG permission and freshness guarantees, GPU token-throughput sizing, and defense-in-depth evaluation and guardrails.

WHAT YOU WILL MASTER IN THIS LESSON
  • Wrap ML models as versioned microservices backed by online feature stores, confidence thresholds, rule-based fallbacks, and human-in-the-loop feedback logs.
  • Design multi-stage candidate generation, ranking, and re-ranking funnels that score 10M items within a 200 ms latency budget without training-serving skew.
  • Size vector indexes (31 GB for 10M 768-d vectors), hybrid lexical-semantic retrieval, and permission-filtered RAG pipelines with event-driven incremental updates.
  • Calculate GPU replica counts from peak output tokens/s (continuous batching, caching, INT8 quantization, model routing) and enforce offline/online evals and safety guardrails.
SYSTEMATIC BUILDING-BLOCK CHECKLIST (FROM MODULE 01)

4-Point Infrastructure Primitive Audit

Check off each dimension as you study this building block—interviewers test what it solves, how it works inside, what it costs, and how it fails.

0 / 4 Checked
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?

5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)

Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL

Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.

✗ NO-HIRE RED FLAG

Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.

2. Right-Sized Design
✓ STRONG HIRE SIGNAL

Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).

✗ NO-HIRE RED FLAG

Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.

3. Technical Depth
✓ STRONG HIRE SIGNAL

Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.

✗ NO-HIRE RED FLAG

Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.

4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL

Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'

✗ NO-HIRE RED FLAG

Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.

5. Communication
✓ STRONG HIRE SIGNAL

Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.

✗ NO-HIRE RED FLAG

Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.

ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 1Section 01: Models as Components in a System
ARCHITECTURE / DATA FLOW DIAGRAM
Reviewerfallback rulesReview serviceGPU · own deploysSpam modelmodel serviceFeature storev13 testingModel registryHuman reviewPrediction logTraining job
SECTION 01 OF 7

1. Models as Components in a System

★ FREE PREVIEW
Key Takeaway: Deploy every ML model as an isolated, versioned microservice surrounded by a low-latency feature store, confidence-band thresholds, deterministic rule fallbacks, and a labeled prediction log.
ARCHITECTURE / DATA FLOW DIAGRAM
Reviewerfallback rulesReview serviceGPU · own deploysSpam modelmodel serviceFeature storev13 testingModel registryHuman reviewPrediction logTraining job

Why Models Run as Dedicated Versioned Services

Consider a review platform screening 2,000 newly submitted reviews per second for spam. Rather than embedding multi-gigabyte model weights as an in-process library inside the CPU-bound Review Service, the application calls a separate Spam Model Service that returns a probability score in [0.0, 1.0]. Service isolation decouples GPU/accelerator scaling and memory footprints from web tiers and lets ML teams deploy or roll back pinned model artifacts from a Model Registry independently of application releases. Because raw HTTP requests omit historical signals (account_age_days, prior_rejected_reviews_30d), the model service fetches precomputed entity attributes in single-digit milliseconds from an online Feature Store.

Surrounding ComponentArchitectural RoleFailure or Operational Benefit
Model RegistryStores immutable trained weights, hyperparameters, and lineageInstant rollback by switching the serving deployment's pinned version
Online Feature StoreServes low-latency entity features (account age, velocity) by keyKeeps request payloads thin and standardizes feature definitions
Threshold & Fallback PolicyMaps [0, 1] probabilities to actions; runs heuristics on timeoutPrevents GPU brownouts from blocking user review submissions
Prediction Log + Human QueueJoins model scores with human moderator verdictsGenerates high-signal labeled ground truth for the next Training Job

Confidence Thresholds, Fallbacks, and Feedback Loops

Unlike deterministic business logic, model outputs are calibrated probabilities that occasionally err, so callers apply tiered decision thresholds: scores >= 0.90 automatically hide the review, scores in [0.50, 0.90) route uncertain borderline cases to a Human Review queue, and scores < 0.50 publish immediately. If the Model Service times out or crashes, the Review Service degrades gracefully to fast deterministic heuristics—such as holding reviews containing URLs from accounts under 24 hours old—and queues unscored reviews for asynchronous rescoring once GPUs recover. Every prediction and subsequent human decision writes to a Prediction Log, closing the retraining loop.

Interview Signal: Never treat an ML model call as infallible RPC logic in a design interview: always specify (1) what latency timeout triggers fallback, (2) what deterministic heuristic runs when the model is unreachable, and (3) how human overrides feed back into retraining.
SECTION 02 OF 7

2. Ranking, Prediction, and Classification Pipelines

🔒 PRO SECTION
Key Takeaway: Multi-stage recommendation and search funnels pair a cheap high-recall candidate generator with a heavy high-precision neural ranker and deterministic business re-ranking, reducing 500 seconds of naive scoring to 50 ms.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 03 OF 7

3. Tokens, Context Windows, and Generation

🔒 PRO SECTION
Key Takeaway: LLM capacity, latency, and memory scale with subword tokens: prompt prefill runs in parallel, whereas autoregressive decoding emits one token per sequential step, making streaming and context budgeting mandatory.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 04 OF 7

4. Embeddings and Retrieval

🔒 PRO SECTION
Key Takeaway: Dense vector embeddings capture semantic similarity where keyword overlap fails, requiring ~31 GB for 10M 768-d vectors, Approximate Nearest Neighbor (ANN) indexing for millisecond latency, and hybrid lexical fusion for exact terms.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 05 OF 7

5. Retrieval-Augmented Generation

🔒 PRO SECTION
Key Takeaway: RAG grounds LLM answers in dynamically updated, permission-filtered passages retrieved at query time, delivering cited up-to-date responses where weight fine-tuning would be stale and unverifiable.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 06 OF 7

6. Serving Models within Latency and Cost Budgets

🔒 PRO SECTION
Key Takeaway: Size GPU fleets from peak output tokens/s rather than raw RPS: combining continuous batching (210 -> 6 replicas), response caching (6 -> 4 replicas), INT8 quantization, and small-model routing slashes inference cost.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 07 OF 7

7. Evaluating and Protecting AI Features

🔒 PRO SECTION
Key Takeaway: Production AI requires offline golden-set metrics (precision, recall, groundedness) paired with canary A/B rollouts, bidirectional input/output guardrails, least-privilege tool scopes, and continuous drift monitoring.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

ACTIVE RECALL & TRADE-OFF PROBES

System Design Checkpoint Quizzes (1)

CHECKPOINT QUIZ #1

In "Models as Components in a System" (Why Models Run as Dedicated Versioned Services), how does a production-grade design handle "Online Feature Store" (Failure or Operational Benefit)?