Determinism as a debugging tool, as actually built

We inherit systems more often than we start them. This is how we read one: what we look at first, and what each thing tells us about decisions made by people we'll never meet.

A pipeline that produces a different answer on Tuesday than it did on Monday for the same inputs cannot be debugged, only observed. Determinism isn't purity, it's the property that makes an investigation finite.

WHAT MAKES A RUN UNREPEATABLE

Nearly always one of four things: reading the clock, reading whatever's current in another system, unstable ordering, or a random seed nobody pinned. Each one individually looks harmless. Together they mean nobody can reproduce last Tuesday.

The clock is the worst offender because it's everywhere and invisible. A job that filters on "the last thirty days" gives a different answer every time it runs, which is correct behaviour and completely destroys your ability to re-examine a specific run.

PASS TIME IN AS AN INPUT

The fix is unglamorous: the run receives its effective time rather than asking for it, and every dependency on "now" flows from that value. Then re-running with the same time gives the same answer, and a disputed output becomes a re-run rather than an archaeology project.

The same applies to reference data. A run that joins against whatever the customer table says today isn't reproducible either. Either snapshot what you read or record the version you read, so a replay can reconstruct the same view.

The exception handlers were the most honest documentation in the repo. Every branch that swallowed an error marked a spot where registrars had once been paged and somebody had made it stop.

def run(as_of: datetime, ref_version: str, inputs: Iterable[Record]):
    """Same (as_of, ref_version, inputs) -> same output. Always."""
    refs = reference_data.at(ref_version)      # not "current"
    window = as_of - timedelta(days=30)        # not datetime.now()

    # Stable ordering: ties broken explicitly, never by arrival order.
    for record in sorted(inputs, key=lambda r: (r.effective_at, r.id)):
        if record.effective_at < window:
            continue
        yield classify(record, refs)

THE PAYOFF IS IN DISPUTES

This looks like engineering hygiene until the first time someone challenges an output. With a deterministic pipeline you re-run and show your work. Without one you're reasoning about what probably happened, in a conversation where being probably right isn't good enough.

It also makes changes safe. You can run the new version against last month's inputs and diff the results, which turns "we think this is better" into a list of exactly which records changed and why.

The schema told the real story. Three nullable columns added in one migration, all named after an enrolment record field, all populated for precisely the weeks when a withdrawal was backdated after the census had been filed was an open ticket.

WHERE IT GOES WRONG

• Ordering that depends on arrival, so ties resolve differently on each run.
• A disputed number that can only be discussed, never re-derived.
• A job filtering on "the last thirty days", so no run can ever be reproduced.
• Joining against whatever reference data says today, making last month's output unexplainable.

Effective time and data version are inputs. Then a disputed run is a re-run.

THE VERDICT

Better than its reputation. Most of what looked like bad engineering turned out to be a sensible answer to a constraint that had since been lifted, and nobody had gone back to check.