Cardinality as your observability budget: the shape that holds up

A pattern earns its keep by being cheaper to apply than to argue about. This one has survived clinical operations and four other sectors that share none of the same vocabulary, which usually means the shape is real and not a coincidence.

Metrics cost money per unique combination of labels, not per data point. One well-meant label containing an identifier can multiply your bill by a thousand overnight, and the alert that tells you is the invoice.

THE MULTIPLICATION NOBODY SEES

A counter with three labels of ten values each is a thousand time series. Add a fourth label holding a customer ID and you've got a thousand per customer. It's a one-line change, it passes review because it's obviously useful, and the cost shows up a month later attached to nobody's decision.

The insidious part is that it works beautifully at first. In staging, with four test customers, the new label is the best thing in the dashboard. Production has forty thousand.

TWO DIFFERENT TOOLS

Metrics are for bounded questions: how many, how slow, how often, sliced by things with few values. Traces and logs are for unbounded ones: what happened to this specific request, for this specific customer. Trying to make metrics answer the second kind is what causes the bill.

Once you've got that split, the design question is easy. Anything with an identifier in it goes on a span or a log line, where you pay per event rather than per combination.

It paid for itself the first time a referral sat unrouted for eleven days because its owner had left. Because the structure was already there, recovery was a query instead of an investigation, and care coordinators heard it from the system rather than from a customer.

PUT A LIMIT SOMEWHERE

Review label sets the way you'd review a database index. Someone has to ask what the cardinality of each one is, and "it's an ID" should end the conversation. Better still, enforce it in a wrapper so an unbounded label can't be passed without an explicit override.

And alarm on your own series count. Finding out from the invoice is a slow, expensive feedback loop, and the fix is a five-minute change if you catch it the same week.

At a multi-site outpatient network it turned a recurring escalation into an ordinary state the system could describe. About 9,000 referrals a month went through without a single manual reconciliation, which hadn't been true of any quarter before it.

WHERE IT GOES WRONG

• A label set that's perfect in staging with four test accounts and ruinous in production.
• Using metrics to answer per-request questions that belong in traces.
• No alarm on series count, so cardinality problems are found by finance.
• An identifier used as a metric label, discovered when the monitoring bill arrives.

Metrics answer bounded questions. Anything with an ID in it belongs on a span.

WHEN TO REACH FOR IT

It's cheap on day one and expensive to retrofit, which makes it a default rather than a decision. We put it in the first commit and haven't regretted it yet.