Do We Train on Test Data? Purging CIFAR of Near-Duplicates

By Björn Barz, Joachim Denzler. 2019

116 citations recorded by the source corpus.

The CIFAR-10 and CIFAR-100 datasets are two of the most heavily benchmarked datasets in computer vision and are often used to evaluate novel methods and model architectures in the field of deep learning. However, we find that 3.3% and 10% of the images from the test sets of these datasets have duplicates in the training set. These duplicates are easily recognizable by memorization and may, hence, bias the comparison of image recognition techniques regarding their generalization capability. To eliminate this bias, we provide the “fair CIFAR” (ciFAIR) dataset, where we replaced all duplicates in the test sets with new images sampled from the same domain. The training set remains unchanged, in order not to invalidate pre-trained models. We then re-evaluate the classification performance of various popular state-of-the-art CNN architectures on these new test sets to investigate whether recent research has overfitted to memorizing data instead of learning abstract concepts. We find a significant drop in classification accuracy of between 9% and 14% relative to the original performance on the duplicate-free test set. We make both the ciFAIR dataset and pre-trained models publicly available and furthermore maintain a leaderboard for tracking the state of the art.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Are we done with ImageNet? (2020, background): “Several studies have revisited common computer vision benchmarks [16, 17, 18, 19] and, regarding ImageNet [1] speciﬁcally, identiﬁed various sources of bias and noise [20, 21, 22].”
Big Transfer (BiT): General Visual Representation Learning (2019, neutral): “test sets throughout the paper for comparability to previously published results. Note that nearduplicates between training and test sets have previously been reported by [40] for ILSVRC-2012, and by [3] for CIFAR. In Section D in the appendix we present a few duplicates found between the ILSVRC-201”
FSD50K: An Open Dataset of Human-Labeled Sound Events (2020, methodology): “The importance of reliable evaluation sets is highlighted by recent research in computer vision which focuses on improving the evaluation and/or validation sets of widely-used datasets— [43] for CIFAR-10/-100 [44]; and [45, 46] for ImageNet [47].”
Self-Distillation as Instance-Specific Label Smoothing (2020, background): “[3]), but have different yi’s, which could inform us about other labels that could be drawn from zi.”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “For example, [8] ﬁnd that 10% of the images from the CIFAR-100 [76] test set have duplicates in the training set.”
Language Models are Unsupervised Multitask Learners (2019, background): “For instance CIFAR-10 has 3.3% overlap between train and test images (Barz & Denzler, 2019).”
On Provable Copyright Protection for Generative Models (2023, methodology): “We also chose images which did not have duplicates in the CIFAR-10 train test, using data from Barz and Denzler [2020].”
On Memorization in Probabilistic Deep Generative Models (2021, background): “The fact that commonly-used data sets contain numerous (near) duplicates [12] therefore provides one reason for memorization of training observations.”
Large Scale Learning of General Visual Representations for Transfer (2019, result): “Note that near-duplicates between training and test sets have previously been reported by [40] for ILSVRC-2012, and by [3] for CIFAR.”
A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication (2021, result): “A similar study of the incredibly popular CIFAR-10 and CIFAR-100 datasets [51] found that up to 10% of the images from the test set have duplicates in the training set [9].”
Bugs in the Data: How ImageNet Misrepresents Biodiversity (2022, unknown): “Recent research has placed increased scrutiny on many popular datasets, uncovering biases (Prabhu and Birhane 2021), duplicates (Barz and Denzler 2020) and problematic class definitions (Crawford and Paglen 2019).”
Datamodels: Predicting Predictions from Training Data (2022, methodology): “One can use this primitive to identify issues in datasets such as duplicated training examples [LIN+21] or train-test leakage [BD20] (test examples that have near-duplicates in the training set).”