Dashboard/System design/36. Design a Web Crawler

SYSTEM DESIGN · LESSON 36 OF 52 · 12 MIN VIDEO · 1 TOPICS

Design a Web Crawler

Module Progress (0 / 1 Lessons Completed)0%
System design Track (0 / 52 Modules) · Overall 0/80 (0%)0%
⌁

Narrated HD Video Lesson (12 min) · Pro

Lesson Overview + Section #01 + RESHADED Checklist + Quiz #1 are unlocked free below (no login required). Unlock all 80 HD video lessons and deep-dive sections — or watch our two free sample video lessons first.

0% READSec 01/01 · System Design: Web Crawler
SYSTEM DESIGN VIDEO LESSON & GUIDE

Design a Web Crawler

A web crawler traverses the directed graph of the public internet—starting from high-authority seed URLs, downloading HTML documents, extracting outgoing hyperlinks, and feeding search engine inverted indexes, web archives, and LLM pretraining pipelines. Operating at 5 billion pages/month (~1,930 pages/s and 6 PB/year of raw HTML) turns a simple breadth-first traversal loop into a distributed systems problem governed by strict per-host politeness, adversarial crawler traps, and pervasive duplicate content. L4-L6 interviewers evaluate how cleanly you decouple global URL prioritization from per-domain rate limiting inside the URL frontier, bound memory for URL and content deduplication, and partition crawl state across worker nodes without distributed lock contention.

WHAT YOU WILL MASTER IN THIS LESSON
  • Derive steady-state crawl throughput (1,930 pages/s), network ingress (1.54 Gbps), concurrent open sockets (~1,930 in-flight fetches), and multi-year blob storage (6 PB/year) from monthly crawl targets.
  • Architect a Mercator-style dual-queue URL frontier separating global priority front queues from single-worker per-host back queues stored on local SSDs.
  • Bound deduplication memory across 5 billion pages by combining URL canonicalization and a 6 GB in-memory Bloom filter (1% false-positive rate) with 64-bit content fingerprints (40 GB).
  • Scale horizontally without cross-node coordination using host-based consistent hashing, local DNS/robots.txt caching, trap guardrails, and durable frontier checkpoints.
SYSTEMATIC INTERVIEW CHECKLIST · RESHADED + 45-MIN CLOCK

8-Step RESHADED Execution Checklist

Follow this exact sequence from the Intro narration on every system design prompt so you never miss requirements, math, contracts, or the 13-minute deep dive.

0 / 8 Checked
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?

5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)

Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL

Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.

✗ NO-HIRE RED FLAG

Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.

2. Right-Sized Design
✓ STRONG HIRE SIGNAL

Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).

✗ NO-HIRE RED FLAG

Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.

3. Technical Depth
✓ STRONG HIRE SIGNAL

Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.

✗ NO-HIRE RED FLAG

Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.

4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL

Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'

✗ NO-HIRE RED FLAG

Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.

5. Communication
✓ STRONG HIRE SIGNAL

Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.

✗ NO-HIRE RED FLAG

Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.

ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 5Section 01: System Design: Web Crawler
ARCHITECTURE / DATA FLOW DIAGRAM
Seed URLsknown sitesTo-do listURLs to visitFetcherdownloads pageindex, archivePage storesaved copies40 links foundExtractorfinds links
SECTION 01 OF 1

1. System Design: Web Crawler

★ FREE PREVIEW
Key Takeaway: A web-scale crawler scales a simple fetch-extract-filter loop to 5 billion pages/month by routing URLs via host-consistent hashing, isolating each origin behind a single-worker politeness back queue, and filtering seen URLs and mirror pages with in-memory Bloom filters and 64-bit content fingerprints.
SECTION 01 DIAGRAMS · SLIDE 1 OF 5System Design: Web Crawler
ARCHITECTURE / DATA FLOW DIAGRAM
Seed URLsknown sitesTo-do listURLs to visitFetcherdownloads pageindex, archivePage storesaved copies40 links foundExtractorfinds links

Requirements & Back-of-the-Envelope Capacity Estimation

A search engine or web archive can index only the pages it discovers through continuous breadth-first graph traversal starting from known seed URLs. Although the core control loop (Seed URLs -> URL Frontier -> Fetcher -> Page Store -> Extractor -> URL Filter) is conceptually straightforward, executing 5,000,000,000 pages/month across millions of heterogeneous third-party web servers requires strict per-host rate limiting so small origins do not suffer crawler-induced denial-of-service, alongside fault isolation against malformed HTML and stalled sockets. Assuming 2.6 x 10^6 seconds/month, 100 KB average HTML size, and 1.0 s mean end-to-end HTTP fetch latency (DNS, TCP/TLS handshake, TTFB, body transfer), steady-state throughput reaches 1,930 pages/s (193 MB/s or 1.54 Gbps), requiring ~1,930 concurrent open downloads across 20 active (40 provisioned for 2x peak/failover headroom) I/O-multiplexed fetcher hosts.

  • Functional Requirements: Download pages over HTTP/HTTPS (fetch), persist raw HTML with fetch timestamps (store), parse and canonicalize outgoing hyperlinks (extract), suppress duplicate URLs and mirror bodies (skip dupes), and revisit pages proportional to their change rate (recrawl).
  • Non-Functional Requirements: Scale horizontally across 5B pages/month (scalable), enforce robots.txt and per-host request spacing (polite), survive malformed markup, infinite traps, and unresponsive servers (robust), plug in new parsers such as PDF extractors cleanly (extensible), and resume multi-week crawls after node crashes without restarting from seeds (recoverable).
MetricFormula / AssumptionCalculated ScaleArchitectural Implication
Steady-state crawl rate5,000,000,000 pages / 2,600,000 s1,930 pages/sRequires crawling >= 2,000 distinct hosts concurrently at <= 1 req/s/host
Download bandwidth1,930 pages/s * 100 KB * 8 bits/B193 MB/s (~1.54 Gbps)Easily fits multi-NIC commodity cluster egress/ingress
Monthly page storage5,000,000,000 * 100 KB500 TB / monthWrite-once, bulk-read HTML belongs in compressed Blob Storage, not RDBMS
Annual archive storage500 TB/month * 12 months6 PB / yearTier older crawl snapshots to cold object storage with Zstandard compression
Concurrent fetches in flight1,930 pages/s * 1.0 s mean latency1,930 open connectionsFetchers are network-wait bound, requiring async event loops / coroutines
Fetcher fleet size1,930 in-flight / 100 conns/server20 active (40 with 2x headroom)Modest compute fleet; DNS and origin politeness dominate latency
Interview Signal: Interviewers penalize candidates who store 500 TB/month of HTML inside PostgreSQL or Cassandra rows, or who rely on public recursive DNS resolvers (10-50 ms per lookup at 1,930 QPS): pages are immutable blobs read sequentially by downstream indexers, and high-throughput crawlers must run a local caching DNS resolver honoring record TTLs.

High-Level Pipeline & Dual-Queue URL Frontier

Each architectural stage sits behind a narrow module boundary: the URL Frontier schedules the next URL, asynchronous Fetcher workers resolve hostnames via a local caching DNS Resolver and check cached per-domain rules in the Robots Cache (24 h refresh TTL), downloaded HTML lands in a compressed Blob Store, pluggable Extractor modules parse HTML (or PDF/DOCX via extension plugins) and resolve relative paths into absolute URLs, and the URL Filter drops blocked extensions, disallowed paths, and previously seen URLs before re-enqueuing survivors. Inside the URL Frontier, a naive FIFO queue fails because outgoing links on a single page heavily cluster on the same hostname, firing bursty parallel requests at that server. A two-stage Mercator frontier resolves the conflict between priority and politeness: a Prioritizer scores incoming URLs by domain authority (PageRank) and historical change frequency into High, Medium, and Low Front Queues (sampled with weighted probabilities to prevent starvation), while a Selector routes dequeued URLs by hostname into per-host Back Queues, each drained by exactly 1 worker thread enforcing an adaptive politeness pause (delay = k * last_download_duration).

  • Disk-Backed Frontier Buffering: Holding 200M-500M pending URLs (20-50 GB) exceeds cheap heap budgets, so front and back queues persist as append-only segments on local NVMe SSDs with small in-memory head/tail ring buffers.
  • Politeness Enforcement: Before the first HTTP GET to any host, the fetcher retrieves /robots.txt, caches Allow/Disallow prefixes and Crawl-delay directives for 24 hours, and drops disallowed paths without touching the origin.
  • Concurrency Requirement: Sustaining 1,930 pages/s while limiting each host to <= 1 request/s requires maintaining at least 2,000 simultaneously active per-host back queues.
// Mercator Dual-Queue Frontier & Polite Worker Loop
interface CrawlPipeline {
  // Front queues: weighted priority selection (e.g., High 60%, Med 30%, Low 10%)
  dequeuePriorityBatch(): NormalizedUrl[];

  // Back queues: 1 dedicated FIFO per active hostname, mapped via min-heap on next_allowed_ts
  pollReadyHostQueue(nowMs: number): HostQueue | null;
}

async function runHostWorker(hostQueue: HostQueue) {
  const url = await hostQueue.pop(); // Exclusive single-consumer guarantee for hostQueue.hostname
  const rules = await robotsCache.getOrFetch(url.hostname, { ttlSec: 86400 });
  if (!rules.isAllowed(url.pathname)) return;

  const ip = await dnsCache.resolve(url.hostname);
  const t0 = performance.now();
  const resp = await httpFetch(ip, url, { timeoutMs: 10000, maxBytes: 10 * 1024 * 1024 });
  const elapsedMs = performance.now() - t0;

  await blobStore.put(url, resp.body, { fetchedAt: Date.now() });
  await extractorQueue.publish(url, resp.contentType, resp.body);
  hostQueue.nextAllowedTs = Date.now() + Math.max(rules.crawlDelayMs, 3 * elapsedMs);
}

Two-Layer Deduplication, Trap Defenses & Adaptive Recrawling

Uncontrolled web graphs contain massive URL aliasing and syndicated mirror text, requiring deduplication at both frontier admission (URL dedup) and storage commit (content dedup). Before lookup, every extracted link passes through canonicalization (Example.com:80/a?b=2&a=1#top -> http://example.com/a?a=1&b=2): lowercasing scheme and host, stripping #fragment anchors and default ports, resolving ../ segments, and lexicographically sorting query parameters. Rather than spending 500 GB of RAM on 5B raw 100-byte URLs or 40 GB on 8-byte hashes, an in-memory Bloom Filter uses 9.6 bits/URL (6 GB total) at a 1% false-positive rate—accepting that 1 in 100 unseen URLs is skipped in exchange for lock-free O(1) memory lookups—while 8-byte (64-bit) cryptographic/SimHash page fingerprints (40 GB across 5B pages) prevent storing identical mirror content under distinct URLs. To protect crawl budget from infinite dynamic traps (e.g., /cal/2026/11 -> /cal/2027/02 -> ... where every URL is unique) and keep indexed snapshots fresh, enforce structural depth/host caps alongside history-driven recrawl intervals and conditional HTTP GET headers.

  • Anti-Trap & Malformed-Input Guardrails: Reject URLs exceeding max_length (e.g., 2,048 chars) or max_path_depth (e.g., 15 segments), enforce a per-host page cap per crawl cycle, demote repeating numeric path patterns, apply strict socket timeouts (10 s) plus response size caps (10 MB), and run a fault-tolerant HTML5 parser that skips broken DOM subtrees.
  • Adaptive Recrawl & HTTP 304: Compare consecutive content fingerprints to adjust revisit intervals dynamically (unchanged fingerprint triggers exponential backoff; modified fingerprint shortens revisit interval) and send If-Modified-Since / ETag (If-None-Match) so unchanged origins return header-only 304 Not Modified responses.
Deduplication / Freshness MechanismFootprint or Cadence (5B Scale)Formula / SignalEngineering Trade-Off
Raw canonical URL set500 GB5B URLs * 100 bytesExact membership, but far too expensive to hold in single-node RAM
Fixed-width URL hash set40 GB5B URLs * 8-byte hash12.5x smaller than strings; negligible 64-bit collision risk
In-memory URL Bloom filter6 GB RAM5B URLs * 9.6 bits (1% FP)Fits comfortably in RAM; 1% false positives skip 1% of novel URLs
Content fingerprint index40 GB5B pages * 8-byte fingerprintDetects mirrored/aliased pages after fetch; SimHash catches near-duplicates
Adaptive recrawl tiersHourly / Monthly / YearlyNews homepages / Blogs / ArchivesConcentrates finite 1,930 pages/s budget on high-churn authoritative URLs

Distributed Partitioning, Checkpointing & Design Evaluation

Scaling horizontally across tens of machines requires partitioning frontier state without breaking the single-worker-per-host politeness invariant. A cluster-wide URL Router maps every normalized URL to an owning Crawler Node using Consistent Hashing on hostname: because every URL on example.com hashes to the exact same node, that node's local back queue enforces per-host rate limits with zero cross-node locks or distributed token buckets. When a node's extractor discovers an external link belonging to another hash partition, it batches and streams that URL over RPC to the peer node owning that hostname. Each crawler node periodically flushes a consistent checkpoint—its SSD-backed frontier offsets and 6 GB Bloom filter bitset—to a replicated Checkpoint Store; if Crawler Node 2 fails, consistent hashing redistributes only its virtual ring slices to surviving peers, which hydrate its latest checkpoint and replay at most a few minutes of idempotent fetches rather than restarting a weeks-long crawl from seed URLs.

RequirementArchitectural MechanismOperational Trade-Off / Verification
ScalableMulti-node crawler cluster partitioned via consistent hashing on hostnameAdding/removing a node migrates only 1/N of host queues; extractors forward cross-host links
PoliteSingle-worker per-host back queue + adaptive post-fetch delay + cached robots.txtZero concurrent requests per host; host-hash locality avoids distributed rate-limit coordination
RobustURL length/depth caps, per-host cycle budgets, socket timeouts, max-byte caps, forgiving parserInfinite calendar traps and tarpit servers cannot exhaust worker pools or disk budgets
No duplicatesURL canonicalization + 6 GB Bloom filter (URLs) + 40 GB 8-byte fingerprints (content)Trades 1% missed novel URLs for 83x memory compression vs. raw URL strings
ExtensibleDecoupled Fetcher, Extractor, and URL Filter interfaces routed by MIME typeSupporting PDF or DOCX indexing requires registering a new parser plugin only
RecoverablePeriodic frontier + Bloom filter snapshots to durable storage + consistent-hash failoverCrash recovery loses <= 1 checkpoint interval (minutes) of idempotent fetches
Interview Signal: When an interviewer asks about hot domains (such as wikipedia.org or reddit.com) creating skew under hostname hashing, point out that politeness itself caps per-host fetch throughput to a few requests per second—meaning a single back queue never saturates a node's NIC, though mega-domains hit their per-cycle host quota quickly and excess extracted links can be filtered upstream before RPC forwarding.
ACTIVE RECALL & TRADE-OFF PROBES

System Design Checkpoint Quizzes (1)

CHECKPOINT QUIZ #1

In "System Design: Web Crawler" (Requirements & Back-of-the-Envelope Capacity Estimation), how does a production-grade design handle "Download bandwidth" (Architectural Implication)?