Quality gates at the point of ingestion

A pattern earns its keep by being cheaper to apply than to argue about. This one has survived marketplace operations and four other sectors that share none of the same vocabulary, which usually means the shape is real and not a coincidence.

Bad data gets more expensive the further it travels. Rejecting it at the boundary is unpopular and cheap. Cleaning it up in the warehouse is popular and never finishes.

THE COST CURVE

A malformed record caught at ingestion costs one message to one sender. The same record found in a report costs an investigation, a correction, and a conversation about whether any other numbers are wrong. Same record, two orders of magnitude difference.

Downstream cleaning also can't ever be complete, because you're inferring intent from a corrupted artefact. Upstream you can just ask. That asymmetry is the whole argument, and it's worth making early because the political pressure runs the other way.

REJECT WITHOUT LOSING IT

Rejecting isn't discarding. The record goes somewhere durable and inspectable, with the reason attached and a way to resubmit once it's fixed. A gate that drops things is worse than no gate, because it destroys evidence.

Tell the sender in terms they can act on. "Constraint violation on column 7" produces an email to your team. "Row 412: closing date is before opening date" produces a corrected file.

It paid for itself the first time a suspended seller kept transacting through a second account. Because the structure was already there, recovery was a query instead of an investigation, and trust and safety reviewers heard it from the system rather than from a customer.

WHERE STRICTNESS BACKFIRES

Being strict about things that don't matter is how gates get switched off. Reject on what makes the record unusable: missing identifiers, impossible dates, referential breaks. Tolerate cosmetic variation, and normalise it quietly rather than making it someone's problem.

Watch the rejection rate as a signal about the sender rather than the data. A rate that jumps usually means something changed at their end, and asking is faster than diagnosing.

At a two-sided equipment marketplace it turned a recurring escalation into an ordinary state the system could describe. Close to 80,000 listings a month went through without a single manual reconciliation, which hadn't been true of any quarter before it.

WHERE IT GOES WRONG

• Cleaning up in the warehouse, inferring intent from an artefact that's already corrupted.
• A gate that drops rejected records, destroying the evidence needed to fix them.
• Rejection messages written in database vocabulary, so the sender can't act on them.
• Strictness about cosmetic formatting, giving everyone a reason to disable the gate.

Reject at the boundary, keep what you rejected, and say why in the sender's language.

WHEN TO REACH FOR IT

It's cheap on day one and expensive to retrofit, which makes it a default rather than a decision. We put it in the first commit and haven't regretted it yet.