What field service taught us about an append-only log as the source of truth

This one starts in a room with schedulers and a whiteboard covered in arrows. They'd built a workable process around a system that fought them at every step, and the workarounds had quietly become the institutional knowledge. Nobody could describe the job without describing the workarounds.

If you only store the current state, every question about how it got there is unanswerable. Keeping the sequence of things that happened and deriving state from it costs more up front and answers questions you'll otherwise pay a consultant to guess at.

WHAT AN UPDATE THROWS AWAY

An UPDATE statement destroys evidence. It replaces a fact with a different fact and keeps no record that the first one existed. In most tables that's fine. In the four or five tables that matter, it means the answer to "why does this say what it says" is permanently unavailable.

Teams work around this with audit tables bolted on beside the real ones. That's the same idea, implemented after the fact, with two problems: the audit table is written by a trigger nobody tests, and it's never quite complete enough to reconstruct anything.

DERIVED STATE IS A CACHE

The mental shift is treating the current-state table as a cache of the log rather than the truth. You can rebuild it, which means you can fix a bug in how state was computed and replay, instead of writing a migration that patches history.

That replay ability is the real return. We've fixed a two-year-old rounding bug by correcting one function and rebuilding, on a system where the alternative would have been a manual reconciliation of several thousand records.

The clearest thing we saw was how schedulers handled a contested work order. On paper it's one step. In practice it's five, three of them over the phone, none of them written down. Which is why nobody could ever explain first-time fix rate to their director.

THE COSTS, HONESTLY

It's more machinery. Two representations to keep in step, a replay path that needs testing, and a schema for events that's genuinely hard to change once they're in production, because the whole point is that you can't rewrite them.

So it's not a default. It earns its place where the history is itself valuable: money, regulated decisions, anything with a dispute process. Apply it to the whole system and you'll spend your budget on ceremony.

Here it showed up as a queue nobody owned. Some 12,000 work orders a month went through it, and schedulers had learned to check it twice a day because the alternative was a technician drove three hours to a job someone had already closed. A better queue wasn't the answer. Making ownership a property of the work order was.

WHERE IT GOES WRONG

• A replay path that was never run in anger and doesn't work when it's finally needed.
• Event sourcing applied to all forty tables, so every trivial change is a two-day exercise.
• An audit table written by an untested trigger, which turns out to be missing exactly the field you need.
• Events with the schema of the current UI, so the log is unreadable after the next redesign.

Keep the sequence for the records people argue about. Derive the rest.

WHAT WE TOOK AWAY

The work shipped and first-time fix rate moved, but the thing we're proudest of is smaller than the system: schedulers stopped keeping a private spreadsheet. That's usually the honest signal that the model finally matches the job.