Scheduled jobs as distributed systems

A pattern earns its keep by being cheaper to apply than to argue about. This one has survived laboratory sample tracking and four other sectors that share none of the same vocabulary, which usually means the shape is real and not a coincidence.

A cron entry looks like the simplest thing in the system and behaves like the least reliable. It has no history, no observability, no idea whether the last run finished, and a schedule that lies twice a year.

OVERLAP AND CLOCK CHANGES

A job scheduled every five minutes that occasionally takes six will eventually run concurrently with itself. If it isn't written for that, and it almost never is, you get double-processing, or a deadlock, in a way that's very hard to attribute afterwards.

Then there's daylight saving, which deletes an hour once a year and repeats one later. A daily job at 2am in a timezone with a spring transition can be skipped entirely, and an hourly one can run twice against the same window. Both of those have cost clients real money in our experience.

NOBODY KNOWS IF IT RAN

The default failure mode is silence. A job that stops running produces no output, no error, and no alert, and it's discovered when someone notices a report is stale, typically several days later, once the gap is expensive to backfill.

The fix is dead man's switch alerting: the job reports completion, and you alarm on the absence of that report. It's inverted from normal monitoring and it's the only thing that catches a job that isn't running at all.

It paid for itself the first time a batch was reported before its confirmatory run had finished. Because the structure was already there, recovery was a query instead of an investigation, and lab managers heard it from the system rather than from a customer.

GIVE EACH RUN AN IDENTITY

The most useful change is treating a run as a first-class record: which window it covered, when it started and finished, what it produced, whether it succeeded. Suddenly you can answer whether last Tuesday's run happened, what it did, and re-run it if not.

That also makes catch-up tractable. When a job's been down for two days, the question is which windows are missing, which is answerable from a table and guesswork otherwise.

At an environmental testing lab it turned a recurring escalation into an ordinary state the system could describe. Around 20,000 samples a month went through without a single manual reconciliation, which hadn't been true of any quarter before it.

WHERE IT GOES WRONG

• No record of which windows a job has covered, making catch-up a matter of guesswork.
• A five-minute job that sometimes takes six, eventually running against itself.
• A daily 2am job skipped entirely on the spring clock change.
• A job that stopped running weeks ago, found when someone noticed a stale report.

Give every run an identity, a window, and an alarm for when it doesn't report.

WHEN TO REACH FOR IT

It's cheap on day one and expensive to retrofit, which makes it a default rather than a decision. We put it in the first commit and haven't regretted it yet.