Lesson Overview + Section #01 + RESHADED Checklist + Quiz #1 are unlocked free below (no login required). Unlock all 80 HD video lessons and deep-dive sections — or watch our two free sample video lessons first.
Designing a metrics monitoring and alerting platform (modeled after Prometheus, Thanos/Mimir, and Datadog) requires ingesting an unceasing stream of 500,000 samples/s (43.2B samples/day) across 5M active time series from 10,000 servers while serving fleet-wide dashboard queries in 1-2 seconds and firing zero-miss pages during catastrophic outages. Unlike logs (discrete events) or traces (request call graphs), metrics are compact numeric streams where compression algorithms, label cardinality bounds, and failure isolation dictate survival at 03:00 AM. L4-L6 interviewers evaluate pull vs. push collection trade-offs, Gorilla delta-of-delta and XOR compression, multi-tier downsampling rollups, alert storm deduplication, and external dead-man's-switch heartbeats.
WHAT YOU WILL MASTER IN THIS LESSON
Combine pull-based HTTP scraping (service discovery, immediate down-detection, collector-paced backpressure) with a push gateway for short-lived batch jobs behind a durable ingestion queue.
Compress raw 16-byte (int64, float64) samples by >11x (~1.4 bytes/sample) using inverted label indexes, 2-hour immutable chunks, delta-of-delta timestamps, and XOR float encoding while bounding label cardinality.
Accelerate multi-month fleet queries by 6x-360x through tiered downsampling (10 s raw -> 1 min min/max/avg/count -> 1 hr rollups in object storage) and precomputed recording rules.
Architect an outage-proof alerting pipeline with pending flap-suppression windows, active-active rules engines, clustered Alertmanager grouping/routing, and an external cross-provider dead-man's switch.
Follow this exact sequence from the Intro narration on every system design prompt so you never miss requirements, math, contracts, or the 13-minute deep dive.
0 / 8 Checked
RRequirements & ScopeMin 2–8
Clarify who the users are, 3–4 core functional flows, explicit non-goals, and target latency/availability SLAs before drawing a single box.
Avoid trap: Still gathering requirements at minute 15 squeezes the deep dive where L5/L6 offers are decided.
EBack-of-the-Envelope MathMin 8–12
Compute ONLY numbers that actively change architectural choices: DAU → peak read/write RPS, daily & 5-year storage, bandwidth (or tokens/sec & KV-cache VRAM for AI).
Avoid trap: Spending 10 minutes multiplying irrelevant byte counts that never influence sharding or caching.
SStorage & Data ModelMin 12–15
Define core tables, primary/partition keys, and access indexes. Separate small structured metadata rows (~1 KB) from bulk binary blobs (S3 + CDN).
Avoid trap: Picking Cassandra or NoSQL by buzzword when 40 writes/s & 3 TB fit cleanly on one indexed SQL primary + replicas.
AAPIs & Wire ContractsMin 15–17
Write 3–4 concrete REST/gRPC/SSE/WebSocket endpoints with request/response shapes, cursor pagination, and Idempotency-Key headers on mutations.
Avoid trap: Hand-waving client-server contracts and discovering missing parameters halfway through the deep dive.
HHigh-Level DesignMin 17–27
Draw the simplest end-to-end working read and write paths first. Every box on the board must trace back to a stated requirement or number.
Avoid trap: Reciting a memorized 12-microservice Kafka mesh before proving why a simpler synchronous path breaks.
DDetailed Design (Deep Dive)Min 27–36
Proactively audit your high-level diagram, name the top 2 bottlenecks (hot partitions, fan-out skew, lock contention, GPU saturation), and go deep on internals.
Avoid trap: Waiting passively for the interviewer to rescue pacing or skimming every box at shallow surface depth.
EEvaluation & Failure ModesMin 36–41
State what each architectural choice COSTS you, and trace graceful degradation when cache, queue, external vendor, or primary DB fails.
Avoid trap: Assuming happy-path uptime or placing slow 3-second external calls inline on revenue-critical write paths.
DDistinctive FeatureMin 41–45
Solve the signature hard problem unique to this prompt (e.g., double-entry ledger idempotency, Geohash dispatch locks, OT/CRDT sync, or SSE token streaming).
Avoid trap: Spending 20 minutes on generic login/CRUD tables and zero time on the core distributed challenge.
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?
5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)
Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL
Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.
✗ NO-HIRE RED FLAG
Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.
2. Right-Sized Design
✓ STRONG HIRE SIGNAL
Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).
✗ NO-HIRE RED FLAG
Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.
3. Technical Depth
✓ STRONG HIRE SIGNAL
Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.
✗ NO-HIRE RED FLAG
Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.
4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL
Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'
✗ NO-HIRE RED FLAG
Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.
5. Communication
✓ STRONG HIRE SIGNAL
Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.
✗ NO-HIRE RED FLAG
Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.
ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 1Section 01: System Design: Metrics Monitoring and Alerting System
ARCHITECTURE / DATA FLOW DIAGRAM
SECTION 01 OF 2
1. System Design: Metrics Monitoring and Alerting System
★ FREE PREVIEW
Key Takeaway: At 500,000 samples/s (43.2B/day), domain-specific compression shrinks 690 GB/day of raw points to 60 GB/day (0.9 TB for 15 days), shifting the primary engineering bottleneck from disk capacity to sustained append throughput, bounded label cardinality, and failure-independent alerting.
ARCHITECTURE / DATA FLOW DIAGRAM
Metrics vs. Logs and Traces: Observability at Scale
When a disk on 1 of 10,000 production servers crosses 90% utilization at 03:00 AM, no human is watching a dashboard. While Logs capture verbose per-event records and Traces follow individual request lifecycles across microservices, Metrics represent numeric measurements sampled at regular intervals (e.g., every 10 s), identified uniquely by a metric_name plus a set of key-value labels such as disk_used_pct{host="web-7", region="europe"}. A metrics platform serves two distinct workloads over this relentless stream: interactive time-range charts for engineers diagnosing system behavior, and automated continuous rule evaluations (disk_used_pct > 90% for 5m) that page the on-call rotation before capacity exhaustion triggers user-facing downtime.
Time-Series Identity: Every unique combination of metric_name and label=value pairs defines a distinct time series.
Write-Dominated Workload: 99%+ of operations are chronological appends per series with zero in-place mutations, paired with reads skewed heavily toward the most recent hours.
Failure Independence Requirement: Because monitoring is needed most while production clusters are failing, it cannot share databases, networks, or failure domains with the infrastructure it watches.
Functional and Non-Functional Requirements
The system must ingest metrics across bare-metal hosts, containers, databases, load balancers, and custom application counters, retain fine-grained recent samples alongside multi-year historical trends, and route alerts by severity without drowning engineers in duplicate pages. Crucially, sub-second read freshness is non-essential: a dashboard lagging real time by 10-20 s is completely acceptable if doing so enables 11x batch chunk compression and rock-solid write durability.
Category
Requirement
Target & Operational Constraint
Functional
Multi-Source Collection
Ingest infrastructure and application metrics every 10 s across 10,000+ targets.
Functional
Tiered Time-Series Storage
Retain 10 s raw detail short-term (15 days) and downsampled summaries for 1-2+ years.
Functional
Label-Based Querying
Filter, slice, and aggregate series (avg, sum, p99) by arbitrary label selectors.
Functional
Continuous Alert Evaluation
Evaluate user-defined threshold/rate rules every 30 s with sustained for durations.
Functional
Smart Notification Routing
Deduplicate, group, silence, and route alerts to Pager, Chat, or Email with escalation.
Non-Functional
Sustained Write Scalability
Absorb 500,000+ samples/s linearly without dropping samples during DB restarts.
Non-Functional
Interactive Query Latency
Render fleet-wide 6-hour to 30-day dashboard panels in 1-2 s.
Non-Functional
Zero-Miss Alert Reliability
Stay operational during regional outages and page even if the monitor itself dies.
Back-of-the-Envelope Capacity Estimation
Monitoring 10,000 servers that each expose 500 metrics produces 10,000 * 500 = 5,000,000 (5M) active time series. Scraping each series every 10 s generates 5,000,000 / 10 s = 500,000 samples/s, or 500,000 * 86,400 s = 43.2 billion samples/day. In naive uncompressed form—an 8-byte Unix timestamp plus an 8-byte IEEE-754 float (16 bytes/sample)—raw ingestion consumes 43.2B * 16 B ≈ 691.2 GB/day (~690 GB/day). Applying time-series delta-of-delta and XOR compression shrinks each sample to ~1.4 bytes (>11x smaller), reducing daily footprint to ~60 GB/day and allowing a full 15-day window of 10 s raw data to fit inside ~0.9 TB.
Metric
Formula
Estimate
Design Consequence
Active Time Series
10,000 hosts * 500 metrics
5,000,000 series
Inverted label index fits comfortably in distributed TSDB RAM.
Sample Ingestion Rate
5M series / 10 s interval
500,000 samples/s
Rules out row-oriented B-Tree SQL (500K indexed inserts/s drowns).
Daily Sample Count
500K/s * 86,400 s
43.2B samples/day
Mandates append-only per-series memory chunks.
Raw vs. Compressed Bytes
43.2B * 16 B vs. * 1.4 B
690 GB/day -> 60 GB/day
>11x compression makes 15-day hot SSD storage cheap (~0.9 TB).
Interview Signal: Cardinality Explosion Warning: Attaching a high-cardinality label such as user_id (10M users) or request_id creates +10M distinct time series overnight—tripling index memory and stalling ingestion. Restrict labels strictly to low-cardinality dimensions (host, region, service, http_status).
SECTION 02 OF 2
2. Design of a Metrics Monitoring and Alerting System
🔒 PRO SECTION
Key Takeaway: Use consistent-hash pull collectors plus a push gateway feeding a durable queue, store series as Gorilla-compressed 2-hour blocks tiered from NVMe SSD to object storage with 1m/1h rollups, and run redundant alert evaluators watched by an external heartbeat.
Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.
In "System Design: Metrics Monitoring and Alerting System" (Functional and Non-Functional Requirements), how does a production-grade design handle "Functional" (Target & Operational Constraint)?