MixUp as Locally Linear Out-Of-Manifold Regularization

By Hongyu Guo, Yongyi Mao, Richong Zhang. 2018

353 citations recorded by the source corpus.

MixUp (Zhang et al. 2017) is a recently proposed dataaugmentation scheme, which linearly interpolates a random pair of training examples and correspondingly the one-hot representations of their labels. Training deep neural networks with such additional data is shown capable of significantly improving the predictive accuracy of the current art. The power of MixUp, however, is primarily established empirically and its working and effectiveness have not been explained in any depth. In this paper, we develop an understanding for MixUp as a form of “out-of-manifold regularization”, which imposes certain “local linearity” constraints on the model’s input space beyond the data manifold. This analysis enables us to identify a limitation of MixUp, which we call “manifold intrusion”. In a nutshell, manifold intrusion in MixUp is a form of under-fitting resulting from conflicts between the synthetic labels of the mixed-up examples and the labels of original training data. Such a phenomenon usually happens when the parameters controlling the generation of mixing policies are not sufficiently fine-tuned on the training data. To address this issue, we propose a novel adaptive version of MixUp, where the mixing policies are automatically learned from the data using an additional network and objective function designed to avoid manifold intrusion. The proposed regularizer, AdaMixUp, is empirically evaluated on several benchmark datasets. Extensive experiments demonstrate that AdaMixUp improves upon MixUp when applied to the current art of deep classification models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features (2019, background): “Recently, Mixup variants [41, 34, 10, 39] have been proposed; they perform feature-level interpolation and other types of transformations.”
AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty (2019, methodology): “3%, possibly due to an increased chance of manifold intrusion (Guo et al., 2019).”
Albumentations: fast and flexible image augmentations (2018, background): “Cutout [32], Mixup [33], their derivatives [34] and combinations [35].”
How Does Mixup Help With Robustness and Generalization? (2020, background): “, 2018) has been shown to substantially improve generalization and single-step adversarial robustness among a wide rage of tasks, on both supervised (Lamb et al., 2019; Verma et al., 2019a; Guo et al., 2019), and semi-supervised settings (Berthelot et al.”
Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data (2019, background): “There are a few variants of Mixup [17, 41], as well as a recent effort named Cutmix [43] which combined Mixup and Cutout [10] by cutting and pasting patches.”
SODA10M: Towards Large-Scale Object Detection Benchmark for Autonomous Driving (2021, methodology): “Semi-supervised learning methods mainly consist of self training [61, 59] and consistency regularization [46, 65, 20].”
Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity (2021, background): “Guo et al. (2019) trains an additional neural network to optimize a mixing ratio.”
Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup (2020, methodology): “…virtual mixup examples via convex combination or local replacement of data for training (Zhang et al., 2018; Verma et al., 2019; Yun et al., 2019; Guo et al., 2019). foreground objects in vision, prominent syllables in speech, informative textual units in language) (Simonyan et al., 2013;…”
Nonlinear Mixup: Out-Of-Manifold Data Augmentation for Text Classification (2020, methodology): “Research has shown that Mixup and its variants (Zhang et al. 2018; Guo, Mao, and Zhang 2019b) can dramatically improve the predictive accuracy of the current art of deep neural networks.”
On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks (2019, background): “However neither the original work nor any subsequent extensions to mixup [33, 10, 22] have explored the effect of mixup on predictive uncertainty and DNN calibration; this is precisely what we aim to do in this paper.”
FMix: Enhancing Mixed Sample Data Augmentation (2020, methodology): “We summarise previous analyses and theories [53, 29, 9, 12, 45, 51] in Section E of the appendix.”
SeqMix: Augmenting Active Sequence Labeling via Sequence Mixup (2020, background): “Recently, the Mixup variants (Verma et al., 2019; Summers and Dinneen, 2019; Guo et al., 2019b) turn to perform interpolation in the hidden space to capture higher-level information.”
Manifold Mixup: Better Representations by Interpolating Hidden States (2018, methodology): “We include results from (Zhang et al., 2018) † and (Guo et al., 2016) ‡ .”
Second-order Optimization Method for Large Mini-batch: Training ResNet-50 on ImageNet in 35 Epochs (2018, background): “We extend mixup [11, 31] to increase its regularization effect.”