Dashboard/System design/20. Blob Store

SYSTEM DESIGN · LESSON 20 OF 52 · 11 MIN VIDEO · 5 TOPICS

Blob Store

Module Progress (0 / 5 Lessons Completed)0%
System design Track (0 / 52 Modules) · Overall 0/80 (0%)0%
⌁

Narrated HD Video Lesson (11 min) · Pro

Lesson Overview + Section #01 + RESHADED Checklist + Quiz #1 are unlocked free below (no login required). Unlock all 80 HD video lessons and deep-dive sections — or watch our two free sample video lessons first.

0% READSec 01/05 · System Design: A Blob Store
SYSTEM DESIGN VIDEO LESSON & GUIDE

Blob Store

A distributed blob store (Amazon S3, Google Cloud Storage, Azure Blob Storage) persists immutable, unstructured binary objects—videos, photos, backups, and ML checkpoints—across thousands of commodity disk servers at petabyte-to-exabyte scale. At 1.5 PB/day of replicated ingestion and 58 GB/s of read egress, storing large files inside relational databases or single NAS filers collapses under IOPS, cost, and hardware failure rates. L4–L6 interviewers evaluate how candidates separate lightweight control-plane metadata from high-bandwidth data-plane chunk streaming, partition metadata by lexicographic key ranges to support paginated container listing, and combine synchronous rack-diverse replication with asynchronous geo-replication and background garbage collection.

WHAT YOU WILL MASTER IN THIS LESSON
  • Explain why large unstructured binaries belong in a flat-namespace (account/container/blob) object store linked by ID from a metadata database rather than inline DB rows or POSIX trees.
  • Estimate petabyte-scale daily disk growth (1.5 PB/day), sustained ingress/egress bandwidth (46 Gb/s write, 463 Gb/s read), and continuous zero-downtime fleet expansion (~5,500 new data nodes/year).
  • Architect strict control-plane vs. data-plane separation so stateless front ends stream 64 MB chunks directly to storage Data Nodes while committing metadata atomically after replication.
  • Justify range-partitioned metadata indexes over hash sharding to support paginated list_blobs prefix scans alongside dynamic hot-range splitting.
  • Combine synchronous 3x intra-datacenter rack replication, asynchronous cross-region replication, tombstone deletes with background garbage collection, and CDN edge caching.
SYSTEMATIC BUILDING-BLOCK CHECKLIST (FROM MODULE 01)

4-Point Infrastructure Primitive Audit

Check off each dimension as you study this building block—interviewers test what it solves, how it works inside, what it costs, and how it fails.

0 / 4 Checked
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?

5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)

Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL

Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.

✗ NO-HIRE RED FLAG

Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.

2. Right-Sized Design
✓ STRONG HIRE SIGNAL

Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).

✗ NO-HIRE RED FLAG

Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.

3. Technical Depth
✓ STRONG HIRE SIGNAL

Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.

✗ NO-HIRE RED FLAG

Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.

4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL

Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'

✗ NO-HIRE RED FLAG

Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.

5. Communication
✓ STRONG HIRE SIGNAL

Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.

✗ NO-HIRE RED FLAG

Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.

ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 1Section 01: System Design: A Blob Store
ARCHITECTURE / DATA FLOW DIAGRAM
UploadersApp serversvideo 42 → blob IDDatabasetitle, owner, blob IDBlob storethe video bytesViewers
SECTION 01 OF 5

1. System Design: A Blob Store

★ FREE PREVIEW
Key Takeaway: Keep small structured attributes and a blob_id pointer in a database while offloading large immutable byte streams into a flat-namespace distributed blob store fronted by a CDN.
ARCHITECTURE / DATA FLOW DIAGRAM
UploadersApp serversvideo 42 → blob IDDatabasetitle, owner, blob IDBlob storethe video bytesViewers

Why Databases and Single File Servers Fail for Unstructured Media

Suppose a video platform ingests 1,000,000 user clips per day, each tens to hundreds of megabytes. Those payloads are Blobs (Binary Large Objects): opaque byte sequences written once and read many times (WORM), with no internal relational columns to query. Storing gigabyte blobs directly inside PostgreSQL or Cassandra bloats B-tree/LSM pages, evicts hot structured rows from RAM buffer pools, stalls backups and replication logs, and burns expensive low-latency database SSDs on sequential byte storage. Conversely, a single large POSIX file server quickly exhausts disk capacity, becomes a single point of failure, and saturates its NIC long before serving millions of concurrent viewers.

Storage TierTarget Data ProfileScalability & BandwidthCost & Failure Profile
Relational / NoSQL DBSmall structured rows (< 4 KB): title, owner_id, blob_idOptimized for indexed transactional queries, not GB streamsVery high $/TB; large blobs degrade buffer cache & replication
Single File Server / NASModerate shared POSIX files with nested directoriesCapped by 1 chassis disk bay & NIC; directory lock contentionSingle point of failure; vertical scale ceiling
Distributed Blob StoreKB to multi-TB immutable objects (videos, photos, backups)Petabyte/exabyte scale across thousands of disk nodes + CDNLowest $/TB on commodity JBOD disks; self-healing multi-copy HA

Metadata-Byte Separation & The Flat Namespace Model

Production architectures cleanly split structured metadata from raw bytes: when a creator uploads a video, application servers write {video_id, owner_id, title, blob_path} into the application database and stream the raw video bytes into the Blob Store, while viewers stream bytes directly from the Blob Store (via a CDN) without touching the application database. Internally, blob stores abandon deep POSIX directory hierarchies—where resolving /a/b/c/d/video.mp4 requires sequential inode lookups and directory locks—in favor of a flat three-part logical address: account / container / blob_name.

  • Simulated folders: Slashes inside blob_name (e.g., 2026/11/clip_42.mp4) are treated as plain characters in a flat string key, turning any path lookup into a single O(1) or O(log N) key lookup regardless of apparent folder depth.
  • Industry standards: Amazon S3 (account / bucket / key), Azure Blob Storage (account / container / blob), and Google Cloud Storage all build on this exact flat-key object abstraction.
SECTION 02 OF 5

2. Requirements of a Blob Store's Design

🔒 PRO SECTION
Key Takeaway: At 5,000,000 uploads/day (100 MB average), a blob store must absorb 1.5 PB/day of 3x-replicated disk writes, serve 58 GB/s of byte-range streaming reads, and onboard ~5,500 new storage servers per year with zero downtime.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 03 OF 5

3. Design of a Blob Store

🔒 PRO SECTION
Key Takeaway: Strict separation between the control plane (Manager + Replicated Metadata) and the data plane (Front-Ends streaming chunks directly to Data Nodes) prevents metadata coordinators from bottlenecking multi-hundred-gigabit byte transfers.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 04 OF 5

4. Design Considerations of a Blob Store

🔒 PRO SECTION
Key Takeaway: Splitting large blobs into 64 MB rack-striped chunks maximizes parallel throughput, range-partitioning metadata keeps list_blobs on a single shard, and pairing synchronous local replication with asynchronous geo-replication balances durability against write latency.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 05 OF 5

5. Evaluation of a Blob Store's Design

🔒 PRO SECTION
Key Takeaway: Rack-diverse replication plus geo-replication achieves 99.99% availability (~53 min/year) and 11-nines-class durability, range-partitioned metadata scales to billions of objects, and direct parallel chunk streaming sustains multi-hundred-gigabit throughput.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

ACTIVE RECALL & TRADE-OFF PROBES

System Design Checkpoint Quizzes (1)

CHECKPOINT QUIZ #1

In "System Design: A Blob Store" (Why Databases and Single File Servers Fail for Unstructured Media), how does a production-grade design handle "Single File Server / NAS" (Cost & Failure Profile)?