AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty

By Dan Hendrycks, Norman Mu, Ekin Dogus Cubuk, Barret Zoph, Justin Gilmer, Balaji Lakshminarayanan. 2019

1,617 citations recorded by the source corpus.

Modern deep neural networks can achieve high accuracy when the training distribution and test distribution are identically distributed, but this assumption is frequently violated in practice. When the train and test distributions are mismatched, accuracy can plummet. Currently there are few techniques that improve robustness to unforeseen data shifts encountered during deployment. In this work, we propose a technique to improve the robustness and uncertainty estimates of image classifiers. We propose AugMix, a data processing technique that is simple to implement, adds limited computational overhead, and helps models withstand unforeseen corruptions. AugMix significantly improves robustness and uncertainty measures on challenging image classification benchmarks, closing the gap between previous methods and the best possible performance in some cases by more than half.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Pretrained Transformers Improve Out-of-Distribution Robustness (2020, background): “One partial explanation is that Transformer models are pretrained on diverse data, and in computer vision, dataset diversity can improve OOD generalization (Hendrycks et al., 2020) and OOD detection (Hendrycks et al., 2019b).”
On Robustness and Transferability of Convolutional Neural Networks (2020, background): “Using better data augmentation can improve performance on data with synthetic noise [29, 43].”
Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “After the models trained on more data, we ﬁnd that DeepAugment (in combination with AugMix [41]) achieves substantial effective robustness ( ρ = 11 .”
The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization (2020, background): “Hendrycks et al., 2020 [14] find that many recent language models are already robust to many forms of distribution shift, while others [37, 10] find that vision models are largely fragile and argue that data augmentation offers one solution.”
Natural Adversarial Examples (2019, methodology): “Although some data augmentation techniques are purported to greatly improve robustness to distribution shifts [30, 54], their lackluster results on IMAGENET-A show they do not improve robustness on some distribution shifts.”
Towards Robust Vision Transformer (2021, methodology): “Same with the augmentations like MixUp [52], AugMix [18], Ran-dAugment [5], patch-wise augmentation also benefit the model robustness.”
Perceiver: General Perception with Iterative Attention (2021, methodology): “On raw audio the Perceiver gets 38.4, better than most Con-vNet models except CNN-14 (Kong et al., 2020) which uses extra AugMix (Hendrycks et al., 2019) and class-balances the data – we hope to incorporate this in future work.”
VOXLINGUA107: A Dataset for Spoken Language Recognition (2020, methodology): “During training, we apply on-the ﬂy data augmentation using AugMix [14], by randomly distorting the training data using a mix of reverberation and noise augmentation.”
Unsolved Problems in ML Safety (2021, background): “New data augmentation techniques [84] and other sources of simulated data could create inputs that are not easy or possible to create naturally.”
Albumentations: fast and flexible image augmentations (2018, background): “Image augmentations have also been shown to improve convergence [7], generalization and robustness on out-of-distribution samples [8,9], and to overall have more advantages compared to other regularization techniques [10].”
Class Gradient Projection For Continual Learning (2022, methodology): “For the augmentation scheme in contrastive learning, we use AugMix [16] as the augmentation.”
Do not Forget to Attend to Uncertainty while Mitigating Catastrophic Forgetting (2021, background): “These activations are obtained by the network’s prediction, which is overconfident for even for the wrong prediction due to mismatch of train-test distribution [11], and thus do not capture the full prediction distribution.”
Robust fine-tuning of zero-shot models (2021, methodology): “special data augmentation (DeVries & Taylor, 2017; Engstrom et al., 2019; Geirhos et al., 2018a; Hendrycks et al., 2020) or adversarially robust models (Madry et al.”
Image Augmentations for GAN Training (2020, methodology): “The augmentation strategies also range from the basic operations like random crop and horizontal flip to more sophisticated handcrafted operations [10, 39, 43, 16], or even the strategies directly learned by the neural network [8, 44].”
A Simple Way to Make Neural Networks Robust Against Diverse Image Corruptions (2020, positive): “2 AugMix [Hendrycks et al., 2020] 77.”
An Overview of Deep Semi-Supervised Learning (2020, background): “Other works [35, 69, 87, 96] have shown that SSL can be useful in learning from noisy labels, where the noisy labels are discarded, and the noisy examples are considered as unlabeled data and used to regularize training using SSL methods.”
Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup (2020, methodology): “Hendrycks et al. (2020) proposed AugMix which performs Input mixup between clean and augmented images to improve robustness against corrupted datasets as well as the generalization performance.”
Image Data Augmentation for Deep Learning: A Survey (2022, methodology): “However, it does not provide an evaluation on the effectiveness of data augmentation for various actual tasks and lacks some newly proposed methods, such as, Cut-Mix [ Yun et al. , 2019 ] , AugMix [ Hendrycks et al. , 2019 ] , GridMask [ Chen et al. , 2020 ] , etc.”