Reading a media production system through cardinality as your observability budget

We inherit systems more often than we start them. This is how we read one: what we look at first, and what each thing tells us about decisions made by people we'll never meet.

Metrics cost money per unique combination of labels, not per data point. One well-meant label containing an identifier can multiply your bill by a thousand overnight, and the alert that tells you is the invoice.

THE MULTIPLICATION NOBODY SEES

A counter with three labels of ten values each is a thousand time series. Add a fourth label holding a customer ID and you've got a thousand per customer. It's a one-line change, it passes review because it's obviously useful, and the cost shows up a month later attached to nobody's decision.

The insidious part is that it works beautifully at first. In staging, with four test customers, the new label is the best thing in the dashboard. Production has forty thousand.

TWO DIFFERENT TOOLS

Metrics are for bounded questions: how many, how slow, how often, sliced by things with few values. Traces and logs are for unbounded ones: what happened to this specific request, for this specific customer. Trying to make metrics answer the second kind is what causes the bill.

Once you've got that split, the design question is easy. Anything with an identifier in it goes on a span or a log line, where you pay per event rather than per combination.

The exception handlers were the most honest documentation in the repo. Every branch that swallowed an error marked a spot where producers had once been paged and somebody had made it stop.

PUT A LIMIT SOMEWHERE

Review label sets the way you'd review a database index. Someone has to ask what the cardinality of each one is, and "it's an ID" should end the conversation. Better still, enforce it in a wrapper so an unbounded label can't be passed without an explicit override.

And alarm on your own series count. Finding out from the invoice is a slow, expensive feedback loop, and the fix is a five-minute change if you catch it the same week.

The schema told the real story. Three nullable columns added in one migration, all named after a delivery package field, all populated for precisely the weeks when a master was published with the wrong audio mix attached was an open ticket.

WHERE IT GOES WRONG

• An identifier used as a metric label, discovered when the monitoring bill arrives.
• A label set that's perfect in staging with four test accounts and ruinous in production.
• Using metrics to answer per-request questions that belong in traces.
• No alarm on series count, so cardinality problems are found by finance.

Metrics answer bounded questions. Anything with an ID in it belongs on a span.

THE VERDICT

Better than its reputation. Most of what looked like bad engineering turned out to be a sensible answer to a constraint that had since been lifted, and nobody had gone back to check.