Are All Training Examples Created Equal? An Empirical Study

By Kailas Vodrahalli, Ke Li, Jitendra Malik. 2018

65 citations recorded by the source corpus.

Modern computer vision algorithms often rely on very large training datasets. However, it is conceivable that a carefully selected subsample of the dataset is sufficient for training. In this paper, we propose a gradient-based importance measure that we use to empirically analyze relative importance of training images in four datasets of varying complexity. We find that in some cases, a small subsample is indeed sufficient for training. For other datasets, however, the relative differences in importance are negligible. These results have important implications for active learning on deep networks. Additionally, our analysis method can be used as a general tool to better understand diversity of training examples in datasets.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Understanding Dataset Difficulty with V-Usable Information (2021, neutral): “While prior literature has acknowledged that not all data instances are equal (Vodrahalli et al., 2018; Swayamdipta et al., 2020), there have been few efforts to estimate dataset difficulty formally and directly.”
Adversarial Filters of Dataset Biases (2020, background): “Moreover, ImageNet has many classes and a skewed distribution of data (Vodrahalli et al., 2018).”
Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics (2020, background): “This urges a closer look at datasets, where not all examples might contribute equally towards learning (Vodrahalli et al., 2018).”
Imbalance Problems in Object Detection: A Review (2019, background): “Another related set of methods addresses image classification problem from the data redundancy perspective [149], [150], [151], [152], [153].”
Learning from others' mistakes: Avoiding dataset biases without modeling them (2020, background): “This is also reminiscent of ﬁndings from Vodrahalli et al. (2018) and Shrivastava et al. (2016): not all training samples contribute equally towards learning and in some cases, a carefully selected subset of the training set is sufﬁcient to match (or surpass) the performance on the whole set.”
Algorithm fairness in artificial intelligence for medicine and healthcare (2023, background): “…such as attributing influential data points that would influence model performance in data-resource allocation, attributing model-performance change to different types of distribution shift, and attributing the contribution of participants in fair federated learning 83,100,244,335,341–343 .”
A Survey of Deep Active Learning (2020, methodology): “GA [225] finds that not all datasets can be trained on a small sub-sample set because the relative difference of sample importance in some datasets is almost negligible; therefore, it is not advisable to blindly use smaller sub-datasets in the AL context.”
Adapting Convolutional Neural Networks for Geographical Domain Shift (2019, background): “In [34], the authors show that the size of a dataset is not as important as the diversity of samples within it.”
Information-Theoretic Measures of Dataset Difficulty (2021, background): “Estimates relying on model training dynamics (Swayamdipta et al., 2020; Toneva et al., 2018), gradient magnitudes (Vodrahalli et al., 2018), or loss magnitudes (Han et al., 2018) are sensitive to factors such as variance due to initial parameterization.”