Design a Job Scheduler

Problem Statement

Design a distributed job scheduler that supports one-time and recurring (cron-like) jobs. The system must guarantee at-least-once execution, handle worker failures with retry and exponential backoff, support job priorities, and scale to millions of scheduled jobs with thousands of concurrent workers.

Requirements

Functional

• Schedule one-time jobs at a specific timestamp or recurring jobs via cron expressions
• Execute jobs on a pool of distributed workers with priority-based ordering
• Retry failed jobs with configurable exponential backoff (max 3 retries by default)
• Provide job status tracking: PENDING, RUNNING, SUCCEEDED, FAILED, RETRYING

Non-Functional

• Accuracy: Jobs fire within 1 second of their scheduled time
• Reliability: At-least-once execution guarantee -- no silently dropped jobs
• Scale: 10M scheduled jobs, 100K jobs/minute execution throughput
• Fault tolerance: System continues operating if individual workers or scheduler nodes fail

Core Architecture

1. Scheduler Service -- Polls the job store every second for jobs whose next_fire_time <= now. Uses a distributed lock (Redis SETNX) to ensure only one scheduler instance picks each job. Enqueues the job into the appropriate priority queue. For recurring jobs, computes the next fire time from the cron expression and updates the store.

2. Priority Queue (Kafka/Redis) -- Three priority tiers: HIGH, MEDIUM, LOW. Workers consume from HIGH first. Kafka topics partitioned by priority level, or Redis sorted sets with priority as score. Provides backpressure: if workers are overwhelmed, jobs wait in the queue rather than being dropped.

3. Worker Pool -- Stateless workers that pull jobs from the queue, execute them (HTTP callback, gRPC call, or shell command), and report results back. Each worker sends heartbeats every 10 seconds. If a worker dies mid-execution, the job is re-enqueued after a visibility timeout (like SQS).

4. Dead Letter Queue and Alerting -- Jobs that exhaust all retries are moved to a DLQ for manual inspection. Alerting fires for: DLQ size exceeding threshold, job execution latency spikes, and worker pool capacity dropping below minimum.

Database Choice

PostgreSQL for the job definitions table: job_id, cron_expression, next_fire_time (indexed), payload, status, retry_count, priority. The next_fire_time index enables efficient polling (WHERE next_fire_time <= NOW() AND status = 'PENDING'). Redis for distributed locks (scheduler leader election), rate limiting, and as an optional fast queue for high-priority jobs. Kafka for the durable job queue when at-least-once delivery is critical.

Key API Endpoints

POST /api/v1/jobs
  -> Body: \{ name: "daily-report", cron: "0 9 * * *", callback_url: "https://...", priority: "HIGH", payload: \{\} \}
  -> Returns: \{ job_id: "J-100", next_fire_time: "2024-01-01T09:00:00Z" \}

GET /api/v1/jobs/\{job_id\}/status
  -> Returns: \{ job_id: "J-100", status: "SUCCEEDED", last_run: "...", next_run: "...", retry_count: 0 \}

DELETE /api/v1/jobs/\{job_id\}
  -> Returns: \{ deleted: true \}

Scaling Insight

The critical scaling challenge is the thundering herd problem at minute boundaries -- many cron jobs fire at :00 seconds (e.g., "every hour" jobs). Solution: add a configurable jitter (0-30 seconds) to each job's fire time. This spreads 100K jobs scheduled for 9:00:00 across the window 9:00:00-9:00:30, reducing peak queue pressure by 30x and preventing worker pool saturation.

Key Tradeoffs

Original DosiBridge notes for Design a Job Scheduler

This section is written specifically for this page, not copied from a general system-design checklist. The useful way to study Design a Job Scheduler is to connect the concept to a production service where growth exposes the first weak assumption in the original design. That framing changes the design discussion from definitions to operating choices.

What makes this topic distinct

• Core Entity: For Design a Job Scheduler, decide the core entity before choosing tools; otherwise the design looks clean but fails when job changes.
• Read/Write Path: For Design a Job Scheduler, decide the read/write path before choosing tools; otherwise the design looks clean but fails when scheduler changes.
• Contention Point: For Design a Job Scheduler, decide the contention point before choosing tools; otherwise the design looks clean but fails when job changes.
• Abuse Path: For Design a Job Scheduler, decide the abuse path before choosing tools; otherwise the design looks clean but fails when scheduler changes.
• Recovery Workflow: For Design a Job Scheduler, decide the recovery workflow before choosing tools; otherwise the design looks clean but fails when job changes.

Design a Job Scheduler correctness boundary

For this problem, separate the user-visible action from background work. The correctness boundary is the smallest state transition that must not be duplicated, lost, or reordered. Naming that boundary makes the solution more original than a stock API/database/cache template.

Design a Job Scheduler scaling bottleneck

The first bottleneck should be described in the language of this system: a hot timeline, a contested seat, a busy wallet ledger, a media-processing queue, or a search index update delay. That bottleneck decides which component deserves the deepest interview discussion.

Production review question

If Design a Job Scheduler suddenly receives 10x traffic, which entity becomes contested first, and what queue, lock, shard, or cache protects that entity without hiding correctness bugs?

Practical metric to track

Track job-scheduler health as a named dashboard or review item: combine one user-facing metric, one saturation metric, and one correctness metric. For Design a Job Scheduler, this keeps the discussion grounded in behavior that can be measured after deployment instead of generic advice that sounds good in an interview but never gets verified.

Sources

• Design a Job Scheduler -- Reference
• Source: System-Design-Overview