Determinism as a debugging tool, from first principles

Most write-ups of this start in the middle. We'll start earlier than strictly necessary, because nine times in ten the confusion comes from a word two people are using differently, not from the mechanism.

A pipeline that produces a different answer on Tuesday than it did on Monday for the same inputs cannot be debugged, only observed. Determinism isn't purity, it's the property that makes an investigation finite.

WHAT MAKES A RUN UNREPEATABLE

Nearly always one of four things: reading the clock, reading whatever's current in another system, unstable ordering, or a random seed nobody pinned. Each one individually looks harmless. Together they mean nobody can reproduce last Tuesday.

The clock is the worst offender because it's everywhere and invisible. A job that filters on "the last thirty days" gives a different answer every time it runs, which is correct behaviour and completely destroys your ability to re-examine a specific run.

PASS TIME IN AS AN INPUT

The fix is unglamorous: the run receives its effective time rather than asking for it, and every dependency on "now" flows from that value. Then re-running with the same time gives the same answer, and a disputed output becomes a re-run rather than an archaeology project.

The same applies to reference data. A run that joins against whatever the customer table says today isn't reproducible either. Either snapshot what you read or record the version you read, so a replay can reconstruct the same view.

Made concrete: a referral packet passes through several pairs of hands, and each of them believes it owns the record. What's above is how you let all of them be right without letting any of them silently overwrite the others.

def run(as_of: datetime, ref_version: str, inputs: Iterable[Record]):
    """Same (as_of, ref_version, inputs) -> same output. Always."""
    refs = reference_data.at(ref_version)      # not "current"
    window = as_of - timedelta(days=30)        # not datetime.now()

    # Stable ordering: ties broken explicitly, never by arrival order.
    for record in sorted(inputs, key=lambda r: (r.effective_at, r.id)):
        if record.effective_at < window:
            continue
        yield classify(record, refs)

THE PAYOFF IS IN DISPUTES

This looks like engineering hygiene until the first time someone challenges an output. With a deterministic pipeline you re-run and show your work. Without one you're reasoning about what probably happened, in a conversation where being probably right isn't good enough.

It also makes changes safe. You can run the new version against last month's inputs and diff the results, which turns "we think this is better" into a list of exactly which records changed and why.

Picture a multi-site outpatient network at about 9,000 referrals a month. The naive version works perfectly in testing and falls over the first day two care coordinators touch the same referral packet in the same second.

WHERE IT GOES WRONG

• Joining against whatever reference data says today, making last month's output unexplainable.
• Ordering that depends on arrival, so ties resolve differently on each run.
• A disputed number that can only be discussed, never re-derived.
• A job filtering on "the last thirty days", so no run can ever be reproduced.

Effective time and data version are inputs. Then a disputed run is a re-run.

THE SHORT VERSION

None of this is advanced. It's ordinary practice that gets skipped when a date is close, and then paid for with interest by whoever's on call.