mixup: Beyond Empirical Risk Minimization
By Hongyi Zhang, Moustapha Cissé, Yann Dauphin, David Lopez-Paz. 2017
12,076 citations recorded by the source corpus.
Large deep neural networks are powerful, but exhibit undesirable behaviors such as memorization and sensitivity to adversarial examples. In this work, we propose mixup, a simple learning principle to alleviate these issues. In essence, mixup trains a neural network on convex combinations of pairs of examples and their labels. By doing so, mixup regularizes the neural network to favor simple linear behavior in-between training examples. Our experiments on the ImageNet-2012, CIFAR-10, CIFAR-100, Google commands and UCI datasets show that mixup improves the generalization of state-of-the-art neural network architectures. We also find that mixup reduces the memorization of corrupt labels, increases the robustness to adversarial examples, and stabilizes the training of generative adversarial networks.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Mohit Bansal
- Yoshua Bengio
- Jianfeng Gao
- Sergey Levine
- Percy Liang
- Dacheng Tao
- Trevor Darrell
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Ming-Hsuan Yang
- Xiaogang Wang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Optimization & Training
- Quoc V. Le
- Yoshua Bengio
- Frank Hutter
- Soham De
- Samuel L. Smith
- Simon Lacoste-Julien
- Andrew Brock
- Barret Zoph
Citations in the corpus
- Unsupervised Data Augmentation for Consistency Training (2019, neutral): “These methods employ mixup [42] on top of simple augmentations such as flipping and cropping; instead, UDA emphasizes on the use of state-of-the-art data augmentations, leading to significantly better results on CIFAR-10 and SVHN.”
- On Robustness and Transferability of Convolutional Neural Networks (2020, background): “003, no weight decay, the classification head initialized to zeros, Mixup [73] with α = 0.”
- Big Transfer (BiT): General Visual Representation Learning (2019, methodology): “eyer , Zhai? ?, Puigcerver, Yung, Gelly, Houlsby test resolution [57]. The latter is well-suited for transfer learning; we include the resolution change during our ne-tuning step. We found that MixUp [67] is not useful for pre-training BiT, likely due to the abundance of data. However, it is sometim”
- Measuring Robustness to Natural Distribution Shifts in Image Classification (2020, methodology): “This category includes 86 models with an explicit robustness intervention such as adversarially robust models [13, 27, 72, 74, 101], models with special data augmentation [20, 28, 34, 41, 100, 108, 113], and models with architecture modifications [115].”
- The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization (2020, background): “The space of augmentations that help robustness includes various types of noise [22, 27, 21], highly unnatural image transformations [10, 38, 39], or compositions of simple image transformations such as Python Imaging Library operations [4, 15].”
- Bag of Tricks for Image Classification with Convolutional Neural Networks (2018, methodology): “Here we consider another augmentation method called mixup [30].”
- Early Convolutions Help Transformers See Better (2021, methodology): “We use AutoAugment [7], mixup [52] (α = 0.”
- Towards Robust Vision Transformer (2021, methodology): “Same with the augmentations like MixUp [52], AugMix [18], Ran-dAugment [5], patch-wise augmentation also benefit the model robustness.”
- Training data-efficient image transformers & distillation through attention (2020, background): “Regularization like Mixup [57] and Cutmix [56] improve performance.”
- Randaugment: Practical automated data augmentation with a reduced search space (2019, methodology): “Data augmentation is a widely used method for generating additional data to improve machine learning systems, for image classification [43, 23, 7, 54], object detection [13], instance segmentation [10], and speech recognition [21, 16, 36].”
- CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features (2019, methodology): “Table 1 gives an overview of Mixup [47], Cutout [3], and CutMix on image classification, weakly supervised localization, and transfer learning to object detection methods.”
- Perceiver IO: A General Architecture for Structured Inputs & Outputs (2021, background): “, 2019) and MixUp (Zhang et al., 2018) regularization, in addition to RandAugment (Cubuk et al.”
- A U-Net Based Discriminator for Generative Adversarial Networks (2020, methodology): “[43] extends [49] by performing interpolation not only in the input layer but also in the intermediate layers.”
- Sharpness-Aware Minimization for Efficiently Improving Generalization (2020, methodology): “, 2018), and mixed sample augmentations (Zhang et al., 2017; Harris et al., 2020).”
- FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence (2020, methodology): “We only mention those components of the SSL algorithm relevant to producing the artificial labels (for example, Virtual Adversarial Training additionally uses entropy minimization [16], MixMatch and ReMixMatch also use MixUp [50], UDA includes additional techniques like training signal annealing, et”
- MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, methodology): “We also use MixUp [47] on its own as a baseline.”