A primer on cardinality as your observability budget
Cardinality as your observability budget from the start, for a room that contains both engineers and the people who'll live with it.
Most write-ups of this start in the middle. We'll start earlier than strictly necessary, because nine times in ten the confusion comes from a word two people are using differently, not from the mechanism.
Metrics cost money per unique combination of labels, not per data point. One well-meant label containing an identifier can multiply your bill by a thousand overnight, and the alert that tells you is the invoice.
THE MULTIPLICATION NOBODY SEES
A counter with three labels of ten values each is a thousand time series. Add a fourth label holding a customer ID and you've got a thousand per customer. It's a one-line change, it passes review because it's obviously useful, and the cost shows up a month later attached to nobody's decision.
The insidious part is that it works beautifully at first. In staging, with four test customers, the new label is the best thing in the dashboard. Production has forty thousand.
TWO DIFFERENT TOOLS
Metrics are for bounded questions: how many, how slow, how often, sliced by things with few values. Traces and logs are for unbounded ones: what happened to this specific request, for this specific customer. Trying to make metrics answer the second kind is what causes the bill.
Once you've got that split, the design question is easy. Anything with an identifier in it goes on a span or a log line, where you pay per event rather than per combination.
Made concrete: a listing passes through several pairs of hands, and each of them believes it owns the record. What's above is how you let all of them be right without letting any of them silently overwrite the others.
PUT A LIMIT SOMEWHERE
Review label sets the way you'd review a database index. Someone has to ask what the cardinality of each one is, and "it's an ID" should end the conversation. Better still, enforce it in a wrapper so an unbounded label can't be passed without an explicit override.
And alarm on your own series count. Finding out from the invoice is a slow, expensive feedback loop, and the fix is a five-minute change if you catch it the same week.
Picture a two-sided equipment marketplace at close to 80,000 listings a month. The naive version works perfectly in testing and falls over the first day two trust and safety reviewers touch the same listing in the same second.
WHERE IT GOES WRONG
- No alarm on series count, so cardinality problems are found by finance.
- An identifier used as a metric label, discovered when the monitoring bill arrives.
- A label set that's perfect in staging with four test accounts and ruinous in production.
- Using metrics to answer per-request questions that belong in traces.
Metrics answer bounded questions. Anything with an ID in it belongs on a span.
THE SHORT VERSION
None of this is advanced. It's ordinary practice that gets skipped when a date is close, and then paid for with interest by whoever's on call.