Synthetic vs Real: Deep Learning on Controlled Noise
By Lu Jiang, Di Huang, Weilong Yang. 2019
3 citations recorded by the source corpus.
Performing controlled experiments on noisy data is essential in thoroughly understanding deep learning across a spectrum of noise levels. Due to the lack of suitable datasets, previous research have only examined deep learning on controlled synthetic noise, and real-world noise has never been systematically studied in a controlled setting. To this end, this paper establishes a benchmark of real-world noisy labels at 10 controlled noise levels. As real-world noise possesses unique properties, to understand the difference, we conduct a large-scale study across a variety of noise levels and types, architectures, methods, and training settings. Our study shows that: (1) Deep Neural Networks (DNNs) generalize much better on real-world noise. (2) DNNs may not learn patterns first on real-world noisy data. (3) When networks are fine-tuned, ImageNet architectures generalize well on noisy data. (4) Real-world noise appears to be less harmful, yet it is more difficult for robust DNN methods to improve. (5) Robust learning methods that work well on synthetic noise may not work as well on real-world noise, and vice versa. We hope our benchmark, as well as our findings, will facilitate deep learning research on noisy data.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Lu Jiang
- Wolfgang Macherey
- Jacob Eisenstein
- Hossein Mobahi
- Yong Cheng
- Behnam Neyshabur
- Pierre Foret
- Ariel Kleiner
In Computer Vision and Pattern Recognition
- Lu Jiang
- Hossein Mobahi
- Behnam Neyshabur
- Wolfgang Macherey
- Jacob Eisenstein
- Yong Cheng
- Ariel Kleiner
- Pierre Foret
In Evaluation & Benchmarks
- Jacob Eisenstein
- Yong Cheng
- Ariel Kleiner
- Behnam Neyshabur
- Wolfgang Macherey
- Hossein Mobahi
- Lu Jiang
Citations in the corpus
- Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, methodology): “Indeed, simply training a model with SAM outperforms all prior methods specifically targeting label noise robustness, with the exception of MentorMix (Jiang et al., 2019).”
- AdvAug: Robust Adversarial Augmentation for Neural Machine Translation (2020, background): “Recently, mixup (Zhang et al., 2018) has become a popular data augmentation technique for semi-supervised learning (Berth-elot et al., 2019) and overcoming real-world noisy data (Jiang et al., 2019).”