Scheduled Jobs That Fail Silently Are Worse Than No Jobs at All

Cron jobs fail quietly: missed runs, overlaps, half-finished work. Practical patterns for making scheduled tasks observable and safe.

Every production system accumulates scheduled jobs: nightly reports, invoice generation, cache warming, data cleanup, backup triggers. And almost every team eventually discovers one that has been failing for weeks — quietly, with no error in sight, because nobody was watching. A broken endpoint pages someone in minutes. A broken cron job just stops doing its work, and you find out when a customer asks where their invoice went.

The uncomfortable truth: a scheduled job that fails silently is worse than not having the job at all, because its existence makes you believe the work is being done.

Why cron fails differently

Request-driven code has a built-in feedback loop. A user hits an error, you see a 500 in your logs, your error tracker fires. Scheduled jobs have none of that. The common failure modes are all invisible by default:

• The job never ran. The server rebooted at 2 a.m., the crontab was lost in a migration, or the scheduler container wasn't redeployed.
• The job ran and crashed. The exception went to a log file nobody tails, or worse, to /dev/null.
• The job ran twice at once. Last night's run was slow, this morning's run started on top of it, and now two processes are mutating the same rows.
• The job half-finished. It processed 4,000 of 10,000 records, died, and tomorrow's run doesn't know where it left off.

None of these produce a user-facing error at the moment they happen. All of them produce damage you discover later.

Monitor for absence, not just errors

Error alerting can't catch a job that never started. The fix is a dead man's switch: the job pings a heartbeat URL or writes a timestamp on every successful completion, and a separate monitor alerts when the ping doesn't arrive within the expected window.

This inverts the problem. Instead of hoping failures announce themselves, you demand that success does — and treat silence as an incident. Hosted heartbeat monitors exist, but even a tiny script that checks last_success_at against a threshold covers most of the risk.

Alert on the absence of success, not just the presence of errors. Silence is a failure mode.

Make overlap impossible, not unlikely

If a job can take longer than its schedule interval, it eventually will. Guard every job with a lock: a flock on a file for single-server setups, or an advisory lock / lock row in your database when multiple hosts might run it. When the lock is held, the new run should exit loudly — logged and counted — not block quietly.

And set a timeout. A job stuck for six hours holding its lock blocks every subsequent run. A hard kill after a sane deadline, followed by an alert, beats an invisible pileup.

Design every run to be resumable

Assume any run can die midway, because eventually one will. That means:

• Idempotent steps. Re-running the job over already-processed data must be safe — use upserts, dedupe keys, or "already done" checks instead of blind inserts.
• Checkpoints for long work. Persist progress (last processed ID, batch cursor) so the next run continues instead of restarting or double-processing.
• Small batches over big bangs. A job that commits every 500 records loses minutes on a crash; one giant transaction loses everything.

The takeaway

Treat scheduled jobs like production endpoints, not background chores. Every job should hold a lock, report success to something that notices silence, log loudly when it skips or fails, and survive being killed halfway. That's a day of work per job at most — far cheaper than explaining to a customer why three weeks of invoices never went out.