Stop doing this: letting the pipeline vary

It's a well-meant mistake. That's exactly why it sticks around. Every team that makes it can explain why, and the explanation holds up right until the system has to survive its second year.

A pipeline that produces a different answer on Tuesday than it did on Monday for the same inputs cannot be debugged, only observed. Determinism isn't purity, it's the property that makes an investigation finite.

WHAT MAKES A RUN UNREPEATABLE

Nearly always one of four things: reading the clock, reading whatever's current in another system, unstable ordering, or a random seed nobody pinned. Each one individually looks harmless. Together they mean nobody can reproduce last Tuesday.

The clock is the worst offender because it's everywhere and invisible. A job that filters on "the last thirty days" gives a different answer every time it runs, which is correct behaviour and completely destroys your ability to re-examine a specific run.

PASS TIME IN AS AN INPUT

The fix is unglamorous: the run receives its effective time rather than asking for it, and every dependency on "now" flows from that value. Then re-running with the same time gives the same answer, and a disputed output becomes a re-run rather than an archaeology project.

The same applies to reference data. A run that joins against whatever the customer table says today isn't reproducible either. Either snapshot what you read or record the version you read, so a replay can reconstruct the same view.

This is where the reconciliation spreadsheet comes from. Somebody built it once to settle an argument. Now north of 600 deliverables a month depends on a file with one author, no tests, and a filename ending in _v4_final.

def run(as_of: datetime, ref_version: str, inputs: Iterable[Record]):
    """Same (as_of, ref_version, inputs) -> same output. Always."""
    refs = reference_data.at(ref_version)      # not "current"
    window = as_of - timedelta(days=30)        # not datetime.now()

    # Stable ordering: ties broken explicitly, never by arrival order.
    for record in sorted(inputs, key=lambda r: (r.effective_at, r.id)):
        if record.effective_at < window:
            continue
        yield classify(record, refs)

THE PAYOFF IS IN DISPUTES

This looks like engineering hygiene until the first time someone challenges an output. With a deterministic pipeline you re-run and show your work. Without one you're reasoning about what probably happened, in a conversation where being probably right isn't good enough.

It also makes changes safe. You can run the new version against last month's inputs and diff the results, which turns "we think this is better" into a list of exactly which records changed and why.

The bill shows up in the handover notes: nine paragraphs on which of two systems to trust for a given delivery package. When a master was published with the wrong audio mix attached, it took two days to work out what had actually happened, in what order.

WHERE IT GOES WRONG

• A job filtering on "the last thirty days", so no run can ever be reproduced.
• Joining against whatever reference data says today, making last month's output unexplainable.
• Ordering that depends on arrival, so ties resolve differently on each run.
• A disputed number that can only be discussed, never re-derived.

Effective time and data version are inputs. Then a disputed run is a re-run.

THE REPLACEMENT

The fix isn't more discipline. Discipline wears off. The fix is a structure where the wrong thing can't be said, so nobody has to remember the rule at 3am.