Why we chose quality gates at the point of ingestion for media production

Written in the form we use internally, because the useful part of a decision record isn't the decision. It's the context that made it reasonable. In two years someone will want to reverse this, and they deserve to know what we knew at the time.

Bad data gets more expensive the further it travels. Rejecting it at the boundary is unpopular and cheap. Cleaning it up in the warehouse is popular and never finishes.

THE COST CURVE

A malformed record caught at ingestion costs one message to one sender. The same record found in a report costs an investigation, a correction, and a conversation about whether any other numbers are wrong. Same record, two orders of magnitude difference.

Downstream cleaning also can't ever be complete, because you're inferring intent from a corrupted artefact. Upstream you can just ask. That asymmetry is the whole argument, and it's worth making early because the political pressure runs the other way.

REJECT WITHOUT LOSING IT

Rejecting isn't discarding. The record goes somewhere durable and inspectable, with the reason attached and a way to resubmit once it's fixed. A gate that drops things is worse than no gate, because it destroys evidence.

Tell the sender in terms they can act on. "Constraint violation on column 7" produces an email to your team. "Row 412: closing date is before opening date" produces a corrected file.

We modelled it at north of 600 deliverables a month and the difference only showed up in the tail. At median load you couldn't tell them apart. At the ninety-ninth percentile, one of them stopped being able to explain itself.

WHERE STRICTNESS BACKFIRES

Being strict about things that don't matter is how gates get switched off. Reject on what makes the record unusable: missing identifiers, impossible dates, referential breaks. Tolerate cosmetic variation, and normalise it quietly rather than making it someone's problem.

Watch the rejection rate as a signal about the sender rather than the data. A rate that jumps usually means something changed at their end, and asking is faster than diagnosing.

The deciding factor was regulatory, not technical. Producers have to be able to reconstruct why a given delivery package was handled the way it was, months later, in front of someone unfriendly. That killed two of the three options on the spot.

WHERE IT GOES WRONG

• A gate that drops rejected records, destroying the evidence needed to fix them.
• Rejection messages written in database vocabulary, so the sender can't act on them.
• Strictness about cosmetic formatting, giving everyone a reason to disable the gate.
• Cleaning up in the warehouse, inferring intent from an artefact that's already corrupted.

Reject at the boundary, keep what you rejected, and say why in the sender's language.

CONSEQUENCES WE ACCEPTED

We took a slower first two months in exchange for a system you can still reason about in year three. On an eighteen-month horizon we'd have chosen differently, and we said so at the time.