Lesson Overview + Section #01 + RESHADED Checklist + Quiz #1 are unlocked free below (no login required). Unlock all 80 HD video lessons and deep-dive sections — or watch our two free sample video lessons first.
A web crawler traverses the directed graph of the public internet—starting from high-authority seed URLs, downloading HTML documents, extracting outgoing hyperlinks, and feeding search engine inverted indexes, web archives, and LLM pretraining pipelines. Operating at 5 billion pages/month (~1,930 pages/s and 6 PB/year of raw HTML) turns a simple breadth-first traversal loop into a distributed systems problem governed by strict per-host politeness, adversarial crawler traps, and pervasive duplicate content. L4-L6 interviewers evaluate how cleanly you decouple global URL prioritization from per-domain rate limiting inside the URL frontier, bound memory for URL and content deduplication, and partition crawl state across worker nodes without distributed lock contention.
WHAT YOU WILL MASTER IN THIS LESSON
Derive steady-state crawl throughput (1,930 pages/s), network ingress (1.54 Gbps), concurrent open sockets (~1,930 in-flight fetches), and multi-year blob storage (6 PB/year) from monthly crawl targets.
Architect a Mercator-style dual-queue URL frontier separating global priority front queues from single-worker per-host back queues stored on local SSDs.
Bound deduplication memory across 5 billion pages by combining URL canonicalization and a 6 GB in-memory Bloom filter (1% false-positive rate) with 64-bit content fingerprints (40 GB).
Scale horizontally without cross-node coordination using host-based consistent hashing, local DNS/robots.txt caching, trap guardrails, and durable frontier checkpoints.
Follow this exact sequence from the Intro narration on every system design prompt so you never miss requirements, math, contracts, or the 13-minute deep dive.
0 / 8 Checked
RRequirements & ScopeMin 2–8
Clarify who the users are, 3–4 core functional flows, explicit non-goals, and target latency/availability SLAs before drawing a single box.
Avoid trap: Still gathering requirements at minute 15 squeezes the deep dive where L5/L6 offers are decided.
EBack-of-the-Envelope MathMin 8–12
Compute ONLY numbers that actively change architectural choices: DAU → peak read/write RPS, daily & 5-year storage, bandwidth (or tokens/sec & KV-cache VRAM for AI).
Avoid trap: Spending 10 minutes multiplying irrelevant byte counts that never influence sharding or caching.
SStorage & Data ModelMin 12–15
Define core tables, primary/partition keys, and access indexes. Separate small structured metadata rows (~1 KB) from bulk binary blobs (S3 + CDN).
Avoid trap: Picking Cassandra or NoSQL by buzzword when 40 writes/s & 3 TB fit cleanly on one indexed SQL primary + replicas.
AAPIs & Wire ContractsMin 15–17
Write 3–4 concrete REST/gRPC/SSE/WebSocket endpoints with request/response shapes, cursor pagination, and Idempotency-Key headers on mutations.
Avoid trap: Hand-waving client-server contracts and discovering missing parameters halfway through the deep dive.
HHigh-Level DesignMin 17–27
Draw the simplest end-to-end working read and write paths first. Every box on the board must trace back to a stated requirement or number.
Avoid trap: Reciting a memorized 12-microservice Kafka mesh before proving why a simpler synchronous path breaks.
DDetailed Design (Deep Dive)Min 27–36
Proactively audit your high-level diagram, name the top 2 bottlenecks (hot partitions, fan-out skew, lock contention, GPU saturation), and go deep on internals.
Avoid trap: Waiting passively for the interviewer to rescue pacing or skimming every box at shallow surface depth.
EEvaluation & Failure ModesMin 36–41
State what each architectural choice COSTS you, and trace graceful degradation when cache, queue, external vendor, or primary DB fails.
Avoid trap: Assuming happy-path uptime or placing slow 3-second external calls inline on revenue-critical write paths.
DDistinctive FeatureMin 41–45
Solve the signature hard problem unique to this prompt (e.g., double-entry ledger idempotency, Geohash dispatch locks, OT/CRDT sync, or SSE token streaming).
Avoid trap: Spending 20 minutes on generic login/CRUD tables and zero time on the core distributed challenge.
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?
5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)
Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL
Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.
✗ NO-HIRE RED FLAG
Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.
2. Right-Sized Design
✓ STRONG HIRE SIGNAL
Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).
✗ NO-HIRE RED FLAG
Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.
3. Technical Depth
✓ STRONG HIRE SIGNAL
Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.
✗ NO-HIRE RED FLAG
Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.
4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL
Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'
✗ NO-HIRE RED FLAG
Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.
5. Communication
✓ STRONG HIRE SIGNAL
Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.
✗ NO-HIRE RED FLAG
Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.
ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 5Section 01: System Design: Web Crawler
ARCHITECTURE / DATA FLOW DIAGRAM
SECTION 01 OF 1
1. System Design: Web Crawler
★ FREE PREVIEW
Key Takeaway: A web-scale crawler scales a simple fetch-extract-filter loop to 5 billion pages/month by routing URLs via host-consistent hashing, isolating each origin behind a single-worker politeness back queue, and filtering seen URLs and mirror pages with in-memory Bloom filters and 64-bit content fingerprints.
SECTION 01 DIAGRAMS · SLIDE 1 OF 5System Design: Web Crawler
A search engine or web archive can index only the pages it discovers through continuous breadth-first graph traversal starting from known seed URLs. Although the core control loop (Seed URLs -> URL Frontier -> Fetcher -> Page Store -> Extractor -> URL Filter) is conceptually straightforward, executing 5,000,000,000 pages/month across millions of heterogeneous third-party web servers requires strict per-host rate limiting so small origins do not suffer crawler-induced denial-of-service, alongside fault isolation against malformed HTML and stalled sockets. Assuming 2.6 x 10^6 seconds/month, 100 KB average HTML size, and 1.0 s mean end-to-end HTTP fetch latency (DNS, TCP/TLS handshake, TTFB, body transfer), steady-state throughput reaches 1,930 pages/s (193 MB/s or 1.54 Gbps), requiring ~1,930 concurrent open downloads across 20 active (40 provisioned for 2x peak/failover headroom) I/O-multiplexed fetcher hosts.
Functional Requirements: Download pages over HTTP/HTTPS (fetch), persist raw HTML with fetch timestamps (store), parse and canonicalize outgoing hyperlinks (extract), suppress duplicate URLs and mirror bodies (skip dupes), and revisit pages proportional to their change rate (recrawl).
Non-Functional Requirements: Scale horizontally across 5B pages/month (scalable), enforce robots.txt and per-host request spacing (polite), survive malformed markup, infinite traps, and unresponsive servers (robust), plug in new parsers such as PDF extractors cleanly (extensible), and resume multi-week crawls after node crashes without restarting from seeds (recoverable).
Write-once, bulk-read HTML belongs in compressed Blob Storage, not RDBMS
Annual archive storage
500 TB/month * 12 months
6 PB / year
Tier older crawl snapshots to cold object storage with Zstandard compression
Concurrent fetches in flight
1,930 pages/s * 1.0 s mean latency
1,930 open connections
Fetchers are network-wait bound, requiring async event loops / coroutines
Fetcher fleet size
1,930 in-flight / 100 conns/server
20 active (40 with 2x headroom)
Modest compute fleet; DNS and origin politeness dominate latency
Interview Signal: Interviewers penalize candidates who store 500 TB/month of HTML inside PostgreSQL or Cassandra rows, or who rely on public recursive DNS resolvers (10-50 ms per lookup at 1,930 QPS): pages are immutable blobs read sequentially by downstream indexers, and high-throughput crawlers must run a local caching DNS resolver honoring record TTLs.
High-Level Pipeline & Dual-Queue URL Frontier
Each architectural stage sits behind a narrow module boundary: the URL Frontier schedules the next URL, asynchronous Fetcher workers resolve hostnames via a local caching DNS Resolver and check cached per-domain rules in the Robots Cache (24 h refresh TTL), downloaded HTML lands in a compressed Blob Store, pluggable Extractor modules parse HTML (or PDF/DOCX via extension plugins) and resolve relative paths into absolute URLs, and the URL Filter drops blocked extensions, disallowed paths, and previously seen URLs before re-enqueuing survivors. Inside the URL Frontier, a naive FIFO queue fails because outgoing links on a single page heavily cluster on the same hostname, firing bursty parallel requests at that server. A two-stage Mercator frontier resolves the conflict between priority and politeness: a Prioritizer scores incoming URLs by domain authority (PageRank) and historical change frequency into High, Medium, and LowFront Queues (sampled with weighted probabilities to prevent starvation), while a Selector routes dequeued URLs by hostname into per-host Back Queues, each drained by exactly 1 worker thread enforcing an adaptive politeness pause (delay = k * last_download_duration).
Disk-Backed Frontier Buffering: Holding 200M-500M pending URLs (20-50 GB) exceeds cheap heap budgets, so front and back queues persist as append-only segments on local NVMe SSDs with small in-memory head/tail ring buffers.
Politeness Enforcement: Before the first HTTP GET to any host, the fetcher retrieves /robots.txt, caches Allow/Disallow prefixes and Crawl-delay directives for 24 hours, and drops disallowed paths without touching the origin.
Concurrency Requirement: Sustaining 1,930 pages/s while limiting each host to <= 1 request/s requires maintaining at least 2,000 simultaneously active per-host back queues.
Uncontrolled web graphs contain massive URL aliasing and syndicated mirror text, requiring deduplication at both frontier admission (URL dedup) and storage commit (content dedup). Before lookup, every extracted link passes through canonicalization (Example.com:80/a?b=2&a=1#top -> http://example.com/a?a=1&b=2): lowercasing scheme and host, stripping #fragment anchors and default ports, resolving ../ segments, and lexicographically sorting query parameters. Rather than spending 500 GB of RAM on 5B raw 100-byte URLs or 40 GB on 8-byte hashes, an in-memory Bloom Filter uses 9.6 bits/URL (6 GB total) at a 1% false-positive rate—accepting that 1 in 100 unseen URLs is skipped in exchange for lock-free O(1) memory lookups—while 8-byte (64-bit) cryptographic/SimHash page fingerprints (40 GB across 5B pages) prevent storing identical mirror content under distinct URLs. To protect crawl budget from infinite dynamic traps (e.g., /cal/2026/11 -> /cal/2027/02 -> ... where every URL is unique) and keep indexed snapshots fresh, enforce structural depth/host caps alongside history-driven recrawl intervals and conditional HTTP GET headers.
Anti-Trap & Malformed-Input Guardrails: Reject URLs exceeding max_length (e.g., 2,048 chars) or max_path_depth (e.g., 15 segments), enforce a per-host page cap per crawl cycle, demote repeating numeric path patterns, apply strict socket timeouts (10 s) plus response size caps (10 MB), and run a fault-tolerant HTML5 parser that skips broken DOM subtrees.
Adaptive Recrawl & HTTP 304: Compare consecutive content fingerprints to adjust revisit intervals dynamically (unchanged fingerprint triggers exponential backoff; modified fingerprint shortens revisit interval) and send If-Modified-Since / ETag (If-None-Match) so unchanged origins return header-only 304 Not Modified responses.
Deduplication / Freshness Mechanism
Footprint or Cadence (5B Scale)
Formula / Signal
Engineering Trade-Off
Raw canonical URL set
500 GB
5B URLs * 100 bytes
Exact membership, but far too expensive to hold in single-node RAM
Fixed-width URL hash set
40 GB
5B URLs * 8-byte hash
12.5x smaller than strings; negligible 64-bit collision risk
In-memory URL Bloom filter
6 GB RAM
5B URLs * 9.6 bits (1% FP)
Fits comfortably in RAM; 1% false positives skip 1% of novel URLs
Content fingerprint index
40 GB
5B pages * 8-byte fingerprint
Detects mirrored/aliased pages after fetch; SimHash catches near-duplicates
Adaptive recrawl tiers
Hourly / Monthly / Yearly
News homepages / Blogs / Archives
Concentrates finite 1,930 pages/s budget on high-churn authoritative URLs
Scaling horizontally across tens of machines requires partitioning frontier state without breaking the single-worker-per-host politeness invariant. A cluster-wide URL Router maps every normalized URL to an owning Crawler Node using Consistent Hashing on hostname: because every URL on example.com hashes to the exact same node, that node's local back queue enforces per-host rate limits with zero cross-node locks or distributed token buckets. When a node's extractor discovers an external link belonging to another hash partition, it batches and streams that URL over RPC to the peer node owning that hostname. Each crawler node periodically flushes a consistent checkpoint—its SSD-backed frontier offsets and 6 GB Bloom filter bitset—to a replicated Checkpoint Store; if Crawler Node 2 fails, consistent hashing redistributes only its virtual ring slices to surviving peers, which hydrate its latest checkpoint and replay at most a few minutes of idempotent fetches rather than restarting a weeks-long crawl from seed URLs.
Requirement
Architectural Mechanism
Operational Trade-Off / Verification
Scalable
Multi-node crawler cluster partitioned via consistent hashing on hostname
Adding/removing a node migrates only 1/N of host queues; extractors forward cross-host links
Interview Signal: When an interviewer asks about hot domains (such as wikipedia.org or reddit.com) creating skew under hostname hashing, point out that politeness itself caps per-host fetch throughput to a few requests per second—meaning a single back queue never saturates a node's NIC, though mega-domains hit their per-cycle host quota quickly and excess extracted links can be filtered upstream before RPC forwarding.
ACTIVE RECALL & TRADE-OFF PROBES
System Design Checkpoint Quizzes (1)
CHECKPOINT QUIZ #1
In "System Design: Web Crawler" (Requirements & Back-of-the-Envelope Capacity Estimation), how does a production-grade design handle "Download bandwidth" (Architectural Implication)?