Manifold Mixup: Encouraging Meaningful On-Manifold Interpolation as a Regularizer

By Vikas Verma, Alex Lamb, Christopher Beckham, Aaron C. Courville, Ioannis Mitliagkas, Yoshua Bengio. 2018

67 citations recorded by the source corpus.

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, background): “…and applying image-speciﬁc transformations like cropping to inputs, GANs (Salimans et al., 2016; Wei et al., 2018) are very difﬁcult to train on text due to its discrete nature, and mixup (Zhang et al., 2018; Verma et al., 2018) requires a way of smoothly interpolating be-tween different inputs.”
CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features (2019, methodology): “Manifold Mixup [40] applies Mixup operation on the randomly chosen internal feature map.”
MixMatch: A Holistic Approach to Semi-Supervised Learning (2019, methodology): “MixUp can be seen as encouraging the model to have strictly linear behavior “between” examples, by requiring that the model’s output for a convex combination of two inputs is close to the convex combination of the output for each individual input [43, 44, 18].”
Shakedrop Regularization for Deep Residual Learning (2018, methodology): “Widely used regularization methods include data augmentation [18], stochastic gradient descent (SGD) [37], weight decay [20], batch normalization (BN) [15], label smoothing [28], adversarial training [6], mixup [29]–[31], [38], and dropout [26], [32].”
Learning Data Augmentation Strategies for Object Detection (2019, methodology): “We carried out several experiments with Input Mixup [52], Manifold Mixup [46] and Dropblock [13].”
SpeechMix - Augmenting Deep Sound Recognition Using Hidden Space Interpolations (2020, methodology): “Mixup has been demonstrated to work well on image data [10, 11, 12, 13] and text classification [14, 15].”
On Adversarial Mixup Resynthesis (2019, methodology): “These mixing functions could consist of continuous interpolations between latent vectors such as in Verma et al. (2018), genetically-inspired recombination such as crossover, or even a deep neural network which learns the mixing operation.”
MaxUp: A Simple Way to Improve Generalization of Neural Network Training (2020, unknown): “5 Manifold Mixup (Verma et al., 2019) 77.”
On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks (2019, background): “So far we have seen that mixup consistently leads to better calibrated networks compared to the base case, in addition to improving classification performance as has been observed in a number of works [33, 10, 22].”
Training Neural Networks with Local Error Signals (2019, result): “Note that better results have been reported using regularization and data augmentation techniques (Verma et al., 2018; Cubuk et al., 2018).”
Augment Your Batch: Improving Generalization Through Instance Repetition (2020, methodology): “We were interested to see interaction between results obtained with BA together with recent regularization methods such as label-smoothing [33], mixup [43] and manifold-mixup [36].”
Advances in adversarial attacks and defenses in computer vision: A survey (2021, background): “original model, this practice also helps in robustness against adversarial samples [352].”
Stiffness: A New Perspective on Generalization in Neural Networks (2019, methodology): “Our observations seem connected to recent work on regularization using interpolation between data points in Verma et al. (2018).”
Nonlinear Mixup: Out-Of-Manifold Data Augmentation for Text Classification (2020, methodology): “Unlike existing Mixup methods (Zhang et al. 2018; Guo, Mao, and Zhang 2019b; Verma et al. 2018) where the input pairs and label pairs share the same linear, scalar mixing policy, our approach enables nonlinear interpolation policy in the form of a matrix to interpolate both the input pairs and label”
Semi-Supervised Learning by Label Gradient Alignment (2019, methodology): “Verma et al. (2018) use WRN-28-2, but we do not include their results in Table 1 because they augment the model with additional regularization such as dropout (Srivastava et al., 2014).”