Post-mortem: feature flags with an expiry date

The alert fired at a bad hour, as they do. The system wasn't down. That was the problem. It was up and confidently serving wrong answers, which is a lot worse than an outage because nobody comes looking.

A flag is a temporary fork in your system that becomes permanent by default. Two flags mean four possible systems, and the combinations nobody tests are the ones that produce the incident.

THE COMBINATORIAL PART

Each flag doubles the number of configurations that exist in production. Ten flags is a thousand and twenty-four, and your test suite covers two of them: everything on and everything off. The configuration that actually breaks is some middle state a customer is in and nobody has ever run.

That's the argument for aggressive removal, and it's worth making with the number rather than the principle. Ten flags sounds tidy. A thousand configurations doesn't.

TWO KINDS OF FLAG, DIFFERENT RULES

Release flags exist to decouple deploy from launch. They're temporary by nature and should be deleted within weeks, because their whole purpose expires the moment the feature is fully on.

Operational switches are different: a kill switch for an expensive integration is legitimately permanent, and pretending otherwise means it gets removed in a cleanup and isn't there the day it's needed. Name the two categories differently so nobody has to guess which they're looking at.

The trigger was boring. A referral sat unrouted for eleven days because its owner had left. The system had no way to represent that, so it picked one, and care coordinators spent the next day proving it wrong with exported CSVs.

MAKE REMOVAL AUTOMATIC

Relying on someone to remember doesn't work. What works is an expiry date recorded with the flag and a build that starts failing when it passes. Annoying by design, and much less annoying than the alternative.

The old code path needs deleting too, not just the conditional. A dead branch left behind is a trap for the next person, who'll reasonably assume it still runs and maintain it.

By the time anyone looked, about 9,000 referrals a month had gone through the affected path. Only a slice of it was wrong, and we couldn't tell which slice without a full replay. The replay was the one thing we'd never tested.

WHERE IT GOES WRONG

• Release flags and permanent kill switches sharing one list, so cleanup removes the wrong ones.
• A dead code branch left behind after the flag is removed, maintained by mistake for years.
• Removal depending on someone remembering, which is to say not happening.
• Ten flags, a thousand possible configurations, and two of them tested.

Every release flag gets a death date, enforced by the build.

WHAT CHANGED AFTERWARDS

Two action items survived: the two that removed a decision. Everything on the list that asked someone to be more careful was quietly dead within a quarter, which is roughly what we expected when we wrote it.