Scheduled jobs as distributed systems, as actually built

We inherit systems more often than we start them. This is how we read one: what we look at first, and what each thing tells us about decisions made by people we'll never meet.

A cron entry looks like the simplest thing in the system and behaves like the least reliable. It has no history, no observability, no idea whether the last run finished, and a schedule that lies twice a year.

OVERLAP AND CLOCK CHANGES

A job scheduled every five minutes that occasionally takes six will eventually run concurrently with itself. If it isn't written for that, and it almost never is, you get double-processing, or a deadlock, in a way that's very hard to attribute afterwards.

Then there's daylight saving, which deletes an hour once a year and repeats one later. A daily job at 2am in a timezone with a spring transition can be skipped entirely, and an hourly one can run twice against the same window. Both of those have cost clients real money in our experience.

NOBODY KNOWS IF IT RAN

The default failure mode is silence. A job that stops running produces no output, no error, and no alert, and it's discovered when someone notices a report is stale, typically several days later, once the gap is expensive to backfill.

The fix is dead man's switch alerting: the job reports completion, and you alarm on the absence of that report. It's inverted from normal monitoring and it's the only thing that catches a job that isn't running at all.

The exception handlers were the most honest documentation in the repo. Every branch that swallowed an error marked a spot where schedulers had once been paged and somebody had made it stop.

GIVE EACH RUN AN IDENTITY

The most useful change is treating a run as a first-class record: which window it covered, when it started and finished, what it produced, whether it succeeded. Suddenly you can answer whether last Tuesday's run happened, what it did, and re-run it if not.

That also makes catch-up tractable. When a job's been down for two days, the question is which windows are missing, which is answerable from a table and guesswork otherwise.

The schema told the real story. Three nullable columns added in one migration, all named after a work order field, all populated for precisely the weeks when a technician drove three hours to a job someone had already closed was an open ticket.

WHERE IT GOES WRONG

• A job that stopped running weeks ago, found when someone noticed a stale report.
• No record of which windows a job has covered, making catch-up a matter of guesswork.
• A five-minute job that sometimes takes six, eventually running against itself.
• A daily 2am job skipped entirely on the spring clock change.

Give every run an identity, a window, and an alarm for when it doesn't report.

THE VERDICT

Better than its reputation. Most of what looked like bad engineering turned out to be a sensible answer to a constraint that had since been lifted, and nobody had gone back to check.