The incident that taught us evolving an API without versioning it
Evolving an API without versioning it, learned the expensive way on a laboratory sample tracking system that stayed up while being wrong.
The alert fired at a bad hour, as they do. The system wasn't down. That was the problem. It was up and confidently serving wrong answers, which is a lot worse than an outage because nobody comes looking.
A version number is a promise to maintain two systems. Most changes don't need one. They need additive fields, tolerant readers, and the discipline to never repurpose a name. Save the version bump for the change that genuinely can't be made compatibly.
ADDITIVE IS ALMOST ALWAYS AVAILABLE
New optional field, new endpoint, new value in an enum that clients already handle by ignoring what they don't recognise. The overwhelming majority of real changes fit one of those shapes, and none of them break anybody.
What breaks people is subtler: tightening validation, making an optional field required, changing what a value means while keeping its name. That last one is the worst because nothing fails loudly. The caller carries on, now wrong.
NEVER REPURPOSE A NAME
If the meaning changes, the name changes. `status` meaning something new in March than it did in February is a bug you've distributed to every consumer, and it will be found by a customer rather than a test.
Deprecate by addition. Add the new field, populate both, tell consumers, then measure whether anyone still reads the old one. That last part needs instrumentation, which is the step that makes removal possible rather than theoretical.
The trigger was boring. A batch was reported before its confirmatory run had finished. The system had no way to represent that, so it picked one, and lab managers spent the next day proving it wrong with exported CSVs.
WHEN YOU DO HAVE TO BREAK IT
Sometimes the model was wrong and no amount of additive change fixes it. Then version, and version the whole surface rather than one endpoint, because a client shouldn't have to track which parts are on which version.
And set the retirement date when you launch the new one, with the old one emitting a warning your consumers can see. A version with no end date isn't a migration, it's a second product.
By the time anyone looked, around 20,000 samples a month had gone through the affected path. Only a slice of it was wrong, and we couldn't tell which slice without a full replay. The replay was the one thing we'd never tested.
WHERE IT GOES WRONG
- A v1 that never dies, because nothing forced the conversation.
- Reusing a field name with a new meaning, so consumers keep working and quietly go wrong.
- Making an optional field required, and finding out from a customer's integration.
- Deprecating an endpoint with no instrumentation, so nobody can prove it's safe to remove.
Add, don't repurpose. If the meaning changes, so does the name.
WHAT CHANGED AFTERWARDS
Two action items survived: the two that removed a decision. Everything on the list that asked someone to be more careful was quietly dead within a quarter, which is roughly what we expected when we wrote it.