The incident that taught us choosing a tenant isolation model

The alert fired at a bad hour, as they do. The system wasn't down. That was the problem. It was up and confidently serving wrong answers, which is a lot worse than an outage because nobody comes looking.

Tenant isolation is one of the few decisions that's genuinely hard to change later, because it reaches into every query and every backup. Pick it deliberately, on the basis of what happens when you get it wrong, not on what's fastest to start.

THE THREE OPTIONS

A shared table with a tenant column is cheapest to build and cheapest to operate, and it puts one WHERE clause between your clients and each other's data. A schema per tenant costs more operationally and makes the isolation structural. A database per tenant is expensive and unambiguous.

There's no universally right answer. There's a right answer given how many tenants you expect, how different their data volumes are, and what the consequence of a leak actually is. Ten enterprise clients with audit requirements and ten thousand self-serve accounts want different answers.

THE FORGOTTEN WHERE CLAUSE

With shared tables, the whole model rests on every query being filtered, forever, by everyone. That's not a thing humans do reliably. So it can't be a convention. It has to be enforced somewhere that a new engineer can't route around, whether that's row-level security in the database or a query layer that won't build an unscoped statement.

Test it adversarially. Write a test that tries to read another tenant's row through every access path you have, and treat a pass as a load-bearing part of the suite. The one path nobody tested is the one that leaks, and it's usually a report or an export.

The trigger was boring. An approval was issued against a stale valuation. The system had no way to represent that, so it picked one, and underwriters spent the next day proving it wrong with exported CSVs.

THE PARTS PEOPLE FORGET

Isolation doesn't stop at the table. It applies to backups. Can you restore one tenant without the others? To exports, to logs, to error reports, to caches keyed carelessly. Each of those has leaked somewhere, and the cache key is the one that catches good teams.

Also plan for the tenant that's a hundred times larger than the rest. In a shared model, their data volume becomes everyone's query planner problem, and that conversation arrives with no warning.

By the time anyone looked, around 2,500 applications a month had gone through the affected path. Only a slice of it was wrong, and we couldn't tell which slice without a full replay. The replay was the one thing we'd never tested.

WHERE IT GOES WRONG

• One tenant a hundred times bigger than the rest, quietly ruining everyone's query plans.
• A missing WHERE clause in a report path that nobody tested adversarially.
• Cache keys without the tenant in them, serving one client's data to another.
• No way to restore a single tenant, so every recovery is all-or-nothing.

Isolation has to be structural. A convention that everyone must remember isn't isolation.

WHAT CHANGED AFTERWARDS

Two action items survived: the two that removed a decision. Everything on the list that asked someone to be more careful was quietly dead within a quarter, which is roughly what we expected when we wrote it.