Why we chose scheduled jobs as distributed systems for public sector casework

Written in the form we use internally, because the useful part of a decision record isn't the decision. It's the context that made it reasonable. In two years someone will want to reverse this, and they deserve to know what we knew at the time.

A cron entry looks like the simplest thing in the system and behaves like the least reliable. It has no history, no observability, no idea whether the last run finished, and a schedule that lies twice a year.

OVERLAP AND CLOCK CHANGES

A job scheduled every five minutes that occasionally takes six will eventually run concurrently with itself. If it isn't written for that, and it almost never is, you get double-processing, or a deadlock, in a way that's very hard to attribute afterwards.

Then there's daylight saving, which deletes an hour once a year and repeats one later. A daily job at 2am in a timezone with a spring transition can be skipped entirely, and an hourly one can run twice against the same window. Both of those have cost clients real money in our experience.

NOBODY KNOWS IF IT RAN

The default failure mode is silence. A job that stops running produces no output, no error, and no alert, and it's discovered when someone notices a report is stale, typically several days later, once the gap is expensive to backfill.

The fix is dead man's switch alerting: the job reports completion, and you alarm on the absence of that report. It's inverted from normal monitoring and it's the only thing that catches a job that isn't running at all.

We modelled it at roughly 6,000 open cases and the difference only showed up in the tail. At median load you couldn't tell them apart. At the ninety-ninth percentile, one of them stopped being able to explain itself.

GIVE EACH RUN AN IDENTITY

The most useful change is treating a run as a first-class record: which window it covered, when it started and finished, what it produced, whether it succeeded. Suddenly you can answer whether last Tuesday's run happened, what it did, and re-run it if not.

That also makes catch-up tractable. When a job's been down for two days, the question is which windows are missing, which is answerable from a table and guesswork otherwise.

The deciding factor was regulatory, not technical. Caseworkers have to be able to reconstruct why a given case record was handled the way it was, months later, in front of someone unfriendly. That killed two of the three options on the spot.

WHERE IT GOES WRONG

• A five-minute job that sometimes takes six, eventually running against itself.
• A daily 2am job skipped entirely on the spring clock change.
• A job that stopped running weeks ago, found when someone noticed a stale report.
• No record of which windows a job has covered, making catch-up a matter of guesswork.

Give every run an identity, a window, and an alarm for when it doesn't report.

CONSEQUENCES WE ACCEPTED

We took a slower first two months in exchange for a system you can still reason about in year three. On an eighteen-month horizon we'd have chosen differently, and we said so at the time.