Dashboard/System design/23. Distributed Task Scheduler

SYSTEM DESIGN · LESSON 23 OF 52 · 11 MIN VIDEO · 5 TOPICS

Distributed Task Scheduler

Module Progress (0 / 5 Lessons Completed)0%
System design Track (0 / 52 Modules) · Overall 0/80 (0%)0%
⌁

Narrated HD Video Lesson (11 min) · Pro

Lesson Overview + Section #01 + RESHADED Checklist + Quiz #1 are unlocked free below (no login required). Unlock all 80 HD video lessons and deep-dive sections — or watch our two free sample video lessons first.

0% READSec 01/05 · System Design: The Distributed Task Scheduler
SYSTEM DESIGN VIDEO LESSON & GUIDE

Distributed Task Scheduler

A distributed task scheduler accepts hundreds of millions of asynchronous jobs per day (~5,800 tasks/s average, ~11,600 tasks/s peak), matches each job's CPU and memory requirements to available worker capacity, and guarantees execution despite machine crashes. In L4-L6 interviews, candidates are evaluated on synchronous persistence before acknowledgment, atomic state-machine claims, starvation-free priority scheduling with multi-tenant quotas, and lease-based fault recovery paired with idempotent execution.

WHAT YOU WILL MASTER IN THIS LESSON
  • Apply Little's Law to size concurrent active CPU cores (57,870 cores), peak worker fleet size (~3,600 servers), and sharded task metadata storage (15 TB).
  • Architect a decoupled pipeline separating synchronous database persistence, priority batching, partitioned queueing, stateless scheduling, and executor pools.
  • Prevent low-priority starvation via priority aging and isolate noisy tenants via submission rate limiters and resource quotas.
  • Combine time-bounded execution leases, worker heartbeats, dead-letter queues, and idempotent task semantics to achieve safe at-least-once execution.
SYSTEMATIC BUILDING-BLOCK CHECKLIST (FROM MODULE 01)

4-Point Infrastructure Primitive Audit

Check off each dimension as you study this building block—interviewers test what it solves, how it works inside, what it costs, and how it fails.

0 / 4 Checked
INTERVIEWER SCORECARD · WHAT MAKES A STRONG HIRE?

5 Signals Bar Raisers Grade (L4 vs. L5 Senior vs. L6 Staff)

Senior (L5 / E5 / Senior SDE) Bar — Autonomous Pacing, Depth & Proactive Trade-Offs: Owns the 45-minute clock without prompting, derives every component quantitatively, surfaces failure modes (hot shards, stampedes, dual writes) BEFORE being asked, and defends consistency vs. latency trade-offs.
1. Scoping
✓ STRONG HIRE SIGNAL

Clarifies users, read/write skew, p99 latency, consistency bounds, and explicit non-goals in minutes 2–8.

✗ NO-HIRE RED FLAG

Jumps straight into drawing load balancers and microservices at minute 2 without asking a single question.

2. Right-Sized Design
✓ STRONG HIRE SIGNAL

Starts with the simplest working architecture justified by the math (e.g., 1 NVMe SQL primary + replicas + Redis for 40 writes/s & 3 TB).

✗ NO-HIRE RED FLAG

Recites a memorized Kafka + multi-DC Cassandra diagram that contradicts their own traffic calculations.

3. Technical Depth
✓ STRONG HIRE SIGNAL

Drives deep into 1–2 hard bottlenecks (partition keys, hot-key mitigation, transactional outbox, KV-cache batching) at minute 27.

✗ NO-HIRE RED FLAG

Skims across 12 boxes at identical buzzword depth and runs out of time before the deep dive.

4. Explicit Trade-Offs
✓ STRONG HIRE SIGNAL

Uses 3-part reasoning: 'A queue here because SMTP spikes to 3 s; checkout stays <60 ms, and we accept eventual email delay + retry dedup.'

✗ NO-HIRE RED FLAG

Name-drops tools ('We put Kafka and ZooKeeper here') without stating the alternative or operational cost.

5. Communication
✓ STRONG HIRE SIGNAL

Thinks aloud every 30–45s, checks alignment at phase transitions, and treats 'What if X fails?' as a cue for where points are awarded.

✗ NO-HIRE RED FLAG

Goes silent for 5 minutes while drawing or defends a flawed assumption defensively when nudged.

ARCHITECTURE & DATA-FLOW SLIDES · SLIDE 1 OF 1Section 01: System Design: The Distributed Task Scheduler
ARCHITECTURE / DATA FLOW DIAGRAM
wait for minutes?Clientsupload videosone machine: OS schedulerTask schedulerdecides what runs whereMachine 1Machine 2Machine 3
SECTION 01 OF 5

1. System Design: The Distributed Task Scheduler

★ FREE PREVIEW
Key Takeaway: A distributed task scheduler decouples accepting work from executing it, matching resource-annotated tasks to available machines across a cluster while surviving mid-run worker failures.
ARCHITECTURE / DATA FLOW DIAGRAM
wait for minutes?Clientsupload videosone machine: OS schedulerTask schedulerdecides what runs whereMachine 1Machine 2Machine 3

Why Asynchronous Task Scheduling Exists

Consider a video platform where 1,000,000 uploads arrive daily and each video must be transcoded into 5 resolutions—generating 5,000,000 encoding tasks per day (~58 tasks/s), each consuming several minutes of CPU time. Executing transcoding synchronously inside the upload HTTP handler would hold client connections open for minutes, exhaust web-server threads, and lose all progress on any server restart. Instead, the upload handler persists a task specification and returns 202 Accepted in milliseconds, delegating resource matching, execution, and retry management to a distributed task scheduler.

  • Request-Path Decoupling: Converts multi-minute CPU-bound work into a sub-50 ms durable enqueue operation.
  • Fleet-Wide Bin Packing: Generalizes an operating system's single-host process scheduler across thousands of servers with heterogeneous free CPU and RAM.

The Four Core Jobs: Accept, Place, Run, and Report

Every distributed task scheduler executes four lifecycle responsibilities: durably accepting task definitions, locating an executor node with sufficient unallocated resources (CPU, RAM, or GPU), supervising execution across transient hardware failures, and reporting terminal status back to callers.

ResponsibilitySingle-Node OS SchedulerDistributed Task Scheduler
AcceptIn-memory kernel runqueue insertionReplicated database commit before client ACK
PlaceAssigns thread to local CPU coreMatches task CPU/RAM request against cluster-wide Resource Manager
Run & RecoverPreemptive time-slicing; host crash kills all processesLease-based remote execution; dead worker's tasks reschedule elsewhere
ReportProcess exit code / waitpid()Queryable lifecycle state (submitted, queued, running, done, failed)

Designing for Routine Machine Failure

Across a cluster of thousands of commodity servers, disk failures, kernel panics, out-of-memory kills, and network partitions happen daily. Once the scheduler acknowledges a submitted task, losing that work is unacceptable—requiring durable state outside the worker fleet and automated timeout-driven rescheduling whenever a worker disappears mid-execution.

Interview Signal: Clarify upfront with your interviewer whether the system schedules ad-hoc immediate jobs, delayed/cron recurring jobs, or DAG workflows; the core dispatch, lease, and retry engine underpins all three.
SECTION 02 OF 5

2. Requirements of a Distributed Task Scheduler's Design

🔒 PRO SECTION
Key Takeaway: Processing 500M tasks/day at 10 seconds of CPU per task sustains ~5,787 tasks/s, keeps ~57,870 CPU cores continuously busy (~3,600 peak 32-core servers), and generates 15 TB of 30-day metadata.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 03 OF 5

3. Design of a Distributed Task Scheduler

🔒 PRO SECTION
Key Takeaway: Persisting tasks synchronously in a sharded relational database before batch-enqueuing to a distributed queue guarantees atomic state transitions and lets submission, scheduling, and execution scale independently.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 04 OF 5

4. Design Considerations of a Distributed Task Scheduler

🔒 PRO SECTION
Key Takeaway: Separate priority queues with aging guarantee urgent execution without starving background jobs, per-client quotas enforce fairness, and heartbeat leases plus idempotent tasks make crash retries safe.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

SECTION 05 OF 5

5. Evaluation of a Distributed Task Scheduler's Design

🔒 PRO SECTION
Key Takeaway: Keeping all authoritative state in a sharded, replicated database and partitioned queue renders schedulers and executors stateless and replaceable, satisfying availability, durability, scalability, and fairness.

Section 1 + Checkpoint Quiz #1 are unlocked free outside the paywall. Unlock all architecture slides, QPS & sharding trade-off tables, and all 80 narrated HD video lessons with Full Access — or inspect our sample payment system lesson first.

ACTIVE RECALL & TRADE-OFF PROBES

System Design Checkpoint Quizzes (1)

CHECKPOINT QUIZ #1

In "System Design: The Distributed Task Scheduler" (The Four Core Jobs: Accept, Place, Run, and Report), how does a production-grade design handle "Place" (Distributed Task Scheduler)?