Cold Case: The Lost MNIST Digits

By Chhavi Yadav, L. Eon Bottou. 2019

131 citations recorded by the source corpus.

Although the popular MNIST dataset [LeCun et al., 1994] is derived from the NIST database [Grother and Hanaoka, 1995], the precise processing steps for this derivation have been lost to time. We propose a reconstruction that is accurate enough to serve as a replacement for the MNIST dataset, with insignificant changes in accuracy. We trace each MNIST digit to its NIST source and its rich metadata such as writer identifier, partition identifier, etc. We also reconstruct the complete MNIST test set with 60,000 samples instead of the usual 10,000. Since the balance 50,000 were never distributed, they enable us to investigate the impact of twenty-five years of MNIST experiments on the reported testing performances. Our results unambiguously confirm the trends observed by Recht et al. [2018, 2019]: although the misclassification rates are slightly off, classifier ordering and model selection remain broadly reliable. We attribute this phenomenon to the pairing benefits of comparing classifiers on the same digits.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

The Effect of Natural Distribution Shift on Question Answering Models (2020, methodology): “…pressure on the statistical guarantees of the holdout method (Dwork et al., 2015), a series of replication studies established there is no adaptive overﬁtting on popular clas-siﬁcation benchmarks like MNIST (Yadav & Bottou, 2019), CIFAR-10 (Recht et al., 2019), and ImageNet (Recht et al., 2019).”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, methodology): “Recollecting Test Sets Recht et al. (2019) create new test sets for CIFAR and ImageNet by closely following the procedure used by the original datasets authors; Yadav and Bottou (2019) perform similar for MNIST.”
Identifying Statistical Bias in Dataset Replication (2020, background): “Recent dataset replication studies (Recht et al., 2019b;a; Yadav & Bottou, 2019) have generally found little evidence of adaptive overfitting: progress on the original benchmark translates to roughly the same amount (or more) of progress on newly constructed test sets.”
Robust fine-tuning of zero-shot models (2021, background): “In contrast, accuracy on the reference distribution is often a reliable predictor for accuracy under distribution shift [67, 68, 92, 95, 104].”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “However, even decade-long test set reuse has surprisingly resulted in little-to-no overﬁtting on popular benchmarks such as MNIST, CIFAR-10, ImageNet, SQuAD, the Netﬂix Prize, and more than 100 Kaggle classiﬁcation competitions [97, 123–125, 128, 163].”
Domain2Vec: Domain Embedding for Unsupervised Domain Adaptation (2020, methodology): “The foreground images are from MNIST [54], USPS [55], EMNIST [56], KMNIST [57], QMNIST [58], FashionMNIST [59].”
Can Small and Synthetic Benchmarks Drive Modeling Innovation? A Retrospective Study of Question Answering Modeling Approaches (2021, background): “…danger of adaptive overﬁtting due to test set reuse (Dwork et al., 2015), a recent line of work examines whether systems have overﬁt to particular test sets by taking existing systems and testing them on newly-constructed test sets (Recht et al., 2019; Yadav and Bottou, 2019; Miller et al., 2020).”
The Evolution of Out-of-Distribution Robustness Throughout Fine-Tuning (2021, background): “The linear relationship holds across a wide range of models and is well-established for several robustness benchmarks, including image classiﬁcation on both synthetic and natural distribution shifts [32, 47, 33, 41], 2D object detection [36], and question-answer models in NLP [24].”
The Benchmark Lottery (2021, methodology): “In a recent study, Arora and Zhang [2021] argue that empirical studies that are based on creating or using new test sets (e.g. [Recht et al., 2018, Yadav and Bottou, 2019, Recht et al., 2019]), although reassuring in some level, are not always possible especially in datasets concerning rare or…”
Model Similarity Mitigates Test Set Overuse (2019, background): “Also, Yadav and Bottou [20] recently released a new test set for the seminal MNIST task, on which they observed no overfitting.”