Dashboard/System design/46. Design a Data Infrastructure System

SYSTEM DESIGN · LESSON 46 OF 52 · 16 MIN VIDEO · 3 TOPICS

Design a Data Infrastructure System

Module Progress (0 / 3 Lessons Completed)0%
System design Track (0 / 52 Modules) · Overall 0/80 (0%)0%
⌁

Narrated HD Video Lesson (16 min) · Pro

Lesson Overview + Section #01 + RESHADED Checklist + Quiz #1 are unlocked free below (no login required). Unlock all 80 HD video lessons and deep-dive sections — or watch our two free sample video lessons first.

0% READSec 01/03 · System Design: Data Infrastructure for AI/ML Systems
SYSTEM DESIGN VIDEO LESSON & GUIDE

Design a Data Infrastructure System

Production machine learning platforms succeed or fail on two invariants: raw events must be captured immutably so any dataset or model can be replayed deterministically, and every feature must be defined once so offline training and online inference never diverge. At consumer scale—50M daily active users generating 10B events/day (350 MB/s peak ingest and 730 TB/year compressed history)—ad-hoc notebook exports collapse under stale data, label leakage, and training-serving skew. L4–L6 system design interviews evaluate whether candidates decouple storage from batch and stream compute, enforce point-in-time feature joins, and restrict costly real-time streaming strictly to features whose latency sensitivity justifies continuous state.

WHAT YOU WILL MASTER IN THIS LESSON
  • Size a multi-tenant ML data platform across event ingestion (350K events/s peak), compressed lakehouse storage (2 TB/day), and low-latency online feature serving (40 GB in-memory state at < 10 ms).
  • Architect decoupled batch and streaming pipelines over an immutable Bronze/Silver/Gold lakehouse using Apache Iceberg snapshots and unified feature definitions.
  • Eliminate label leakage and training-serving skew via point-in-time (ASOF) offline joins and dual-materialized feature stores.
  • Enforce production data reliability using schema registries, at-least-once ingestion with downstream event_id deduplication, idempotent partition overwrites, and automated quality circuit breakers.
SYSTEMATIC INTERVIEW CHECKLIST · RESHADED + 45-MIN CLOCK

8-Step RESHADED Execution Checklist

Follow this exact sequence from the Intro narration on every system design prompt so you never miss requirements, math, contracts, or the 13-minute deep dive.

0 / 8 Checked
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?

5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)

Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL

Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.

✗ NO-HIRE RED FLAG

Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.

2. Right-Sized Design
✓ STRONG HIRE SIGNAL

Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).

✗ NO-HIRE RED FLAG

Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.

3. Technical Depth
✓ STRONG HIRE SIGNAL

Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.

✗ NO-HIRE RED FLAG

Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.

4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL

Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'

✗ NO-HIRE RED FLAG

Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.

5. Communication
✓ STRONG HIRE SIGNAL

Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.

✗ NO-HIRE RED FLAG

Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.

ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 1Section 01: System Design: Data Infrastructure for AI/ML Systems
ARCHITECTURE / DATA FLOW DIAGRAM
Orders DB10 B events a dayClick logsCataloguea week oldData scientistCSV exportsshared by every teamData platformshared pipelinestraining ≠ liveModelrecommendations
SECTION 01 OF 3

1. System Design: Data Infrastructure for AI/ML Systems

★ FREE PREVIEW
Key Takeaway: Shared ML data infrastructure replaces brittle, one-off CSV pipelines with continuous, replayable pipelines that turn raw operational events into consistent features for both historical training and sub-10 ms live inference.
ARCHITECTURE / DATA FLOW DIAGRAM
Orders DB10 B events a dayClick logsCataloguea week oldData scientistCSV exportsshared by every teamData platformshared pipelinestraining ≠ liveModelrecommendations

Problem Statement: Why Ad-Hoc ML Pipelines Break at Scale

In early-stage systems, data scientists manually export rows from transactional databases (Orders DB), web click logs, and product catalog services into CSV files to join inside local notebooks. By the time a recommendation or fraud model reaches production, the training extract is days old, irreproducible, and too large (10 TB/day raw) to fit on a single machine. Worse, backend engineers re-implement feature calculations in application code for live requests, introducing subtle discrepancies in time zones, null handling, or window boundaries that degrade model accuracy in production. A centralized ML data platform solves this by ingesting every source once, recording lineage across transformations, and sharing canonical feature tables across recommendations, search ranking, and fraud detection.

  • Reproducibility gap: Manual SQL dumps overwrite mutable rows, making it impossible to reconstruct the exact dataset used to train a deployed model.
  • Training-serving skew: Separate offline Python notebooks and online Java/Go microservices compute the same feature differently, feeding live models out-of-distribution inputs.
  • Siloed duplication: Without a shared feature catalog, multiple ML teams independently build and maintain redundant aggregation pipelines over identical click and order streams.
Interview Signal: Interviewers look for candidates who frame ML infrastructure around two core failure modes—training-serving skew and temporal label leakage—rather than treating it as a generic ETL warehouse.

Functional & Non-Functional Requirements

The platform must serve two fundamentally opposing access patterns: massive sequential throughput across months of historical records for GPU training, and single-digit-millisecond point lookups of current entity state during live user requests. Because end-to-end model inference typically carries a p99 < 100 ms SLA, online feature retrieval must complete in < 10 ms so adequate budget remains for network hops and neural network forward passes. Simultaneously, storage and compute must scale independently while enforcing strict schema, privacy, and access governance.

CategoryRequirementEngineering Target & Rationale
FunctionalMulti-Modal IngestionIngest high-velocity event streams (clicks, searches) and periodic relational snapshots or CDC feeds (orders, catalog).
FunctionalImmutable Raw RetentionPersist raw payloads durably and cheaply so historical datasets can be reprocessed whenever feature logic changes.
FunctionalScheduled & Continuous TransformsExecute both scheduled batch aggregations and continuous low-latency stream transformations.
FunctionalDual Feature ServingServe point-in-time historical feature matrices for training and latest entity vectors for online inference.
FunctionalVersioning & LineageBind every trained model artifact to immutable dataset snapshots, feature definitions, and upstream source dependencies.
Non-FunctionalIndependent ScalabilityDecouple object storage capacity from ephemeral batch/stream compute clusters as event volume grows.
Non-FunctionalTiered FreshnessUpdate real-time behavioral signals (e.g., 5-minute click counts) within seconds; refresh slow-moving features daily.
Non-FunctionalLow-Latency Online ReadsDeliver online feature lookups in p99 < 10 ms to fit inside a 100 ms end-to-end prediction budget.
Non-FunctionalFault Tolerance & IdempotencyGuarantee zero silent event loss and ensure failed pipeline retries never double-count records.
Non-FunctionalData GovernanceEnforce schema validation, PII masking, automated data quality gates, and role-based access control.

Back-of-the-Envelope Capacity Estimation

Consider an e-commerce platform with 50M daily active users (DAU), each generating 200 interaction events (impressions, clicks, queries, purchases) per day at an average serialized event size of 1 KB. Daily volume reaches 50M * 200 = 10B events/day, averaging 10B / 86,400 s ≈ 116,000 events/s (116 MB/s) and surging 3x during evening peaks or flash sales to 350,000 events/s (350 MB/s). Converting 10 TB/day of raw payloads into Snappy/Zstandard-compressed columnar Parquet achieves a 5x reduction (2 TB/day, or 730 TB/year), whereas the online feature store only retains the latest 100 numeric features (8 bytes each) per user (50M * 100 * 8 B = 40 GB), fitting comfortably in RAM across a small sharded cluster.

Subsystem MetricFormula / CalculationEstimated Scale
Daily Event Volume50M DAU * 200 events/user10B events/day
Ingestion Event Rate10B / 86,400 s (avg) · 3x surge (peak)116K events/s avg · 350K events/s peak
Ingestion Bandwidth116K * 1 KB (avg) · 350K * 1 KB (peak)116 MB/s avg · 350 MB/s peak
Kafka Partition Count350 MB/s peak / 5 MB/s per partition~70 partitions (provision 96–128 for headroom)
Lakehouse Storage Growth10 TB/day raw / 5x columnar compression2 TB/day (730 TB/year)
Online Feature Footprint50M users * 100 features * 8 bytes40 GB (fits entirely in distributed RAM)
Interview Signal: Notice the asymmetry uncovered by estimation: historical lake storage (730 TB/year) is capacity-bound and scan-heavy, while online serving (40 GB) is latency-bound and tiny—proving why a single database cannot economically serve both.
SECTION 02 OF 3

2. High-Level Design of Scalable Data Infrastructure for AI/ML

🔒 PRO SECTION
Key Takeaway: Decouple durability (Kafka + object storage lakehouse) from compute (Spark for historical scans, Flink for sub-second windows) and materialize features from a single canonical registry into separate offline and online stores.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 03 OF 3

3. Detailed Design of Scalable Data Infrastructure for AI/ML

🔒 PRO SECTION
Key Takeaway: Combine schema-validated at-least-once Bronze ingestion, partitioned Parquet + Apache Iceberg snapshots, point-in-time ASOF joins, and automated quality gates to guarantee zero label leakage and full reproducibility.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

ACTIVE RECALL & TRADE-OFF PROBES

System Design Checkpoint Quizzes (1)

CHECKPOINT QUIZ #1

In "System Design: Data Infrastructure for AI/ML Systems" (Functional & Non-Functional Requirements), how does a production-grade design handle "Functional" (Engineering Target & Rationale)?