Check the data is fit before you consume it
Validating incoming data against an inferred schema and a distribution baseline, before it reaches training or serving, converts a silent input-fitness failure into a visible one at the boundary where it entered.
Do this firstEvery change runs the suite before a human looks at it
This is not determinism. Determinism asks whether processing repeated; this asks whether the input was fit to process at all, and a perfectly repeatable pipeline will reproduce a corrupted feature exactly.
The production evidence is about where the check sits rather than how clever it is. Type, domain, presence and value-count constraints, plus a distance measure between the current batch’s distribution and the last one, caught new-column, missing-feature and wrong-type anomalies at detection rates near or above 97% in a system processing trillions of examples a day. The firing rates are the part worth copying: single digits, with roughly half to two-thirds of fired alerts leading to an actual fix.
The counterevidence sets the ceiling. Averaged across eight detection tools on five real datasets, precision was about 47% and recall about 36%, and no tool dominated. Some errors — a misspelled name, a wrong-but-plausible address — were structurally undetectable by any of them.
The decoy
Stacking every check you can think of. Coverage bought that way costs precision on a measured curve: across eight error-detection tools, taking the union of all of them reached 99% recall at 39% precision on one dataset. Most of what then fires is wrong, and an on-call who learns that stops reading.
Evidence
- Monitoring Data Quality at Scale with Statistical Modeling — IIIDimensionality reduction over multi-metric table health signals combined with exponential-smoothing forecasting, flagging one-step-ahead deviations across tens of thousands of tables where manual review is infeasible. Table-level alerts under this scheme are described as much rarer than per-metric alerts, framed explicitly as reducing alert fatigue.
- Data Validation for Machine Learning — IISchema constraints plus inter-batch distribution distance, run ahead of training and serving. Over 30 days: new-column anomalies detected at about 100% with a 10% firing rate and 65% of fired alerts leading to a fix; missing features about 97% detected, 6% firing, 53% fixed; wrong data types about 98% detected, under 1% firing, 100% fixed. Over 70% of pipelines adopted schema-driven unit tests, which caught a 6% rate of code and schema mismatch.
Counterevidence
- Detecting Data Errors: Where are we and what needs to be done? — IAverage precision across tools was about 47% and average recall about 36%, with no tool dominant. Taking the union of all tools raised recall to 85-99% and severely degraded precision — 99% recall at 39% precision on one dataset. A benefit-based ordering recovered 28% average precision for 3.5% recall loss. Some errors, such as a misspelled name or a wrong-but-plausible address, were structurally undetectable by any tool tested.
Last reviewed 2026-08-19.