Watch the input distribution, and test whether the shift matters
Two-sample testing between live traffic and a reference distribution, run over a pre-trained classifier's outputs rather than raw features, detects distribution shift without labels and outperformed the alternatives tested, particularly when few samples are available.
Do this firstCheck the data is fit before you consume it
Compare what is arriving now against what the model was trained on, and treat a significant difference as a signal in its own right.
Two results shape how to use it. Dimensionality reduction through a pre-trained classifier’s soft predictions beat the other methods compared, including the popular domain-classifier approach, which performed badly below about a hundred samples and only caught up with more. And shift is not automatically harm — the same work separates detecting a shift from judging whether it damages accuracy, which is what keeps this from becoming another alerting channel nobody reads.
A caution the study delivers memorably: applied to a canonical benchmark’s own train/test split, treated as identically distributed for twenty-five years, the detector fired — the digits had been drawn differently between collection batches, at p = 2.7e-10.
The decoy
An accuracy dashboard. It needs labels, and in production the labels arrive late or never — by the time accuracy visibly drops you have been serving the degraded model for however long the label delay is.
Evidence
- Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift — ITwo-sample testing using a pre-trained classifier for dimensionality reduction performed best; the popular domain-classifier approach performed badly below about 100 samples and only caught up with more. Applied to the canonical benchmark's own train/test split, the detector found a real shift traced to how one digit was drawn between collection batches, at p = 2.7e-10 — though its own malignancy test judged it harmless to accuracy.
Last reviewed 2026-08-19.