Distributed Job Scheduling: Cron, Delayed Jobs and Leases

A distributed job scheduler runs work at the right time across many machines: delayed jobs ("send a reminder in 24 hours"), recurring cron jobs ("generate invoices at 2 a.m."), and retries. It has to fire millions of jobs on time, never lose one, and run each one once in effect, even when workers crash mid-job. The core pieces are a durable job store, a time-indexed way to find due jobs, leases so workers can claim jobs safely, and at-least-once execution with idempotent handlers.

Architecture

Finding due jobs efficiently

• Database index on run time: SELECT ... WHERE run_at <= now() AND status = 'scheduled' ORDER BY run_at LIMIT 100 FOR UPDATE SKIP LOCKED. Simple and transactional; it scales well with partitioning and is plenty for many systems. See optimistic vs pessimistic locking.
• Time buckets: group jobs by minute (bucket = run_at rounded to the minute), so the dispatcher reads one bucket at a time, sharded across dispatchers.
• Sorted sets: a Redis sorted set scored by run time gives fast "what's due" queries; durability must be handled carefully. See Redis.
• Delay queues: some brokers support delayed delivery directly, convenient for short delays.
• Hierarchical timing wheels in memory handle huge numbers of near-term timers cheaply.

Leases and at-least-once execution

A worker leases a job: it marks it running with an owner and a lease expiry (for example 60 seconds) and renews the lease with heartbeats while working. If the worker crashes, the lease expires and the job becomes available to another worker. This guarantees the job eventually runs, but it can run more than once: a worker might pause (garbage collection, network partition), lose its lease, and resume after another worker started.

• Make handlers idempotent: send the reminder with a deduplication key, write results with upserts. See idempotency.
• Check the lease on completion, and use fencing tokens when a job writes to external systems that must reject stale workers. See distributed locks.
• Exactly-once execution isn't achievable in general; effectively-once comes from at-least-once plus idempotency. See delivery guarantees.

Retries, failures and dead letters

• Retry failed jobs with exponential backoff and jitter, up to a maximum number of attempts. See retries and timeouts.
• Separate retryable errors (timeouts, 503s) from permanent ones (invalid input), which fail immediately.
• Move jobs that exhaust retries to a dead-letter queue with the error, for inspection and replay.
• Give each job type a timeout and a concurrency limit, so a slow downstream service doesn't consume every worker.

Cron and recurring jobs

• Store the schedule (a cron expression and time zone) and compute the next run time after each execution, inserting the next occurrence as a normal delayed job.
• Ensure one instance per occurrence: a unique key on (schedule_id, scheduled_time) stops two dispatchers creating the same run.
• Decide the missed-run policy after an outage: run once to catch up, run every missed occurrence, or skip.
• Handle daylight-saving changes and month ends explicitly; "2:30 a.m. daily" may not exist or may happen twice.

Avoid a single scheduler machine that's a single point of failure. Run several dispatchers that coordinate through the database's row locks, or elect one active dispatcher with a lease. See leader election.