Data Validation for Machine Learning
https://mlsys.org/Conferences/2019/doc/2019/167.pdf
Method
"Our system is deployed in production as an integral part of TFX - an end-to-end machine learning platform at Google."
Population
"Hundreds of product teams use our system to validate trillions of training and serving examples per day, amounting to several petabytes of data per day."
What it does not show
No counterfactual — there is no parallel population of unvalidated pipelines, so it cannot say what incident rate the validation avoided. Does not quantify the on-call cost of the firing rates it reports, or whether check suites go stale. Says nothing about processing repeatability; its entire scope is input fitness.
Eric Breck, Neoklis Polyzotis, Sudip Roy, Steven Euijong Whang, Martin Zinkevich
Schema constraints plus inter-batch distribution distance, run ahead of training and serving. Over 30 days: new-column anomalies detected at about 100% with a 10% firing rate and 65% of fired alerts leading to a fix; missing features about 97% detected, 6% firing, 53% fixed; wrong data types about 98% detected, under 1% firing, 100% fixed. Over 70% of pipelines adopted schema-driven unit tests, which caught a 6% rate of code and schema mismatch.
Tier II: Measured detection, firing and fix rates over a 30-day window across hundreds of real production pipelines. Observational telemetry, no comparison arm of unvalidated pipelines.