Learning from Noisy Large-Scale Datasets with Minimal Supervision

By Andreas Veit, Neil Gordon Alldrin, Gal Chechik, Ivan Krasin, Abhinav Kumar Gupta, Serge J. Belongie. 2017

514 citations recorded by the source corpus.

We present an approach to effectively use millions of images with noisy annotations in conjunction with a small subset of cleanly-annotated images to learn powerful image representations. One common approach to combine clean and noisy data is to first pre-train a network using the large noisy dataset and then fine-tune with the clean dataset. We show this approach does not fully leverage the information contained in the clean set. Thus, we demonstrate how to use the clean annotations to reduce the noise in the large dataset before fine-tuning the network using both the clean set and the full set with reduced noise. The approach comprises a multi-task network that jointly learns to clean noisy annotations and to accurately classify images. We evaluate our approach on the recently released Open Images dataset, containing ~9 million images, multiple annotations per image and over 6000 unique classes. For the small clean set of annotations we use a quarter of the validation set with ~40k images. Our results demonstrate that the proposed approach clearly outperforms direct fine-tuning across all major categories of classes in the Open Image dataset. Further, our approach is particularly effective for a large number of classes with wide range of noise in annotations (20-80% false positive annotations).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Measuring the Effects of Data Parallelism on Neural Network Training (2018, methodology): “Average precision was proposed for Open Images by Veit et al. (2017).”
Self-Training With Noisy Student Improves ImageNet Classification (2019, neutral): “Self-training / co-training has also been shown to work well for a variety of other tasks including leveraging noisy data [81], semantic segmentation [4], text classification [38, 73].”
The Open Images Dataset V4 (2018, background): “This metric is discussed in detail for Open Images in (Veit et al. 2017).”
Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels (2018, methodology): “[38] suggested using a label cleaning network in parallel with a classification network to achieve more noise-robust prediction.”
Using Trusted Data to Train Deep Networks on Labels Corrupted by Severe Noise (2018, background): “[24], use human-verified labels to train a label cleaning network by estimating the residuals between the noisy and clean labels in a multi-label classification setting.”
Tencent ML-Images: A Large-Scale Multi-Label Image Database for Visual Representation Learning (2019, background): “, [5], [28]) have demonstrated that learning from massive noisy labeled images is still able to show good visual representation.”
Unsupervised label noise modeling and loss correction (2019, background): “, 2015), Conditional Random Fields (Vahdat, 2017), or CNNs (Veit et al., 2017).”
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels (2017, methodology): “More recently, methods utilized additional labels for label cleaning (Veit et al., 2017), knowledge distillation (Li et al.”
Cleaning Crowdsourced Labels Using Oracles For Statistical Classification (2018, background): “[38] use a small sample of clean data to reduce the severity of label noise on the (much larger) noisy dataset.”
L_DMI: An Information-theoretic Noise-robust Loss Function (2019, unknown): “Multi-task frameworks that jointly estimates true labels and learns to classify images are also introduced [19,38,41,45].”
Meta-Weight-Net: Learning an Explicit Mapping For Sample Weighting (2019, methodology): “For handling noisy label issue, multiple methods have been designed by correcting noisy labels to their true ones via a supplemental clean label inference step [11, 14, 57, 13, 21, 1, 15].”
Early-Learning Regularization Prevents Memorization of Noisy Labels (2020, methodology): “We focus our discussion on methods that do not assume the availability of small subsets of training data with clean labels (as opposed, for example, to [16, 34, 41]).”
An Overview of Deep Semi-Supervised Learning (2020, methodology): “In addition to image classification, self-training was also successfully applied to a variety of tasks, such as semantic segmentation [7], text classification [96, 77], machine translation [137, 62, 23, 63] and learning from noisy data [152].”
Model-Agnostic Approaches To Handling Noisy Labels When Training Sound Event Classifiers (2019, background): “Other approaches use noise-robust loss functions to mitigate the effect of label noise [14], or leverage an additional set of curated data, for example to train a label cleaning network in order to reduce the noise of a dataset [15].”
Iterative Learning with Open-set Noisy Labels (2018, background): “Previous work has addressed the noisy label problem explicitly or implicitly in a closed-set setting, via either loss correction or noise model based clean label inferring [22, 29, 37, 38].”