How Does Mixup Help With Robustness and Generalization?

By Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, James Y. Zou. 2020

298 citations recorded by the source corpus.

Mixup is a popular data augmentation technique based on taking convex combinations of pairs of examples and their labels. This simple technique has been shown to substantially improve both the robustness and the generalization of the trained model. However, it is not well-understood why such improvement occurs. In this paper, we provide theoretical analysis to demonstrate how using Mixup in training helps model robustness and generalization. For robustness, we show that minimizing the Mixup loss corresponds to approximately minimizing an upper bound of the adversarial loss. This explains why models obtained by Mixup training exhibits robustness to several kinds of adversarial attacks such as Fast Gradient Sign Method (FGSM). For generalization, we prove that Mixup augmentation corresponds to a specific type of data-adaptive regularization which reduces overfitting. Our analysis provides new insights and a framework to understand Mixup.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiashi Feng
2. Yifan Zhang
3. Sung Ju Hwang
4. Jian Liang
5. Dapeng Hu
6. Jaehong Yoon
7. Ce Zhang
8. Weixin Liang

Citations in the corpus

• Representational Continuity for Unsupervised Continual Learning (2021, background): “The interpolated examples not only augments the past tasks’ instances in the replay buffer but also approximates a regularized loss minimization (Zhang et al., 2021 4).”
• Disparities in dermatology AI performance on a diverse, curated clinical image set (2021, methodology): “Mixup generates synthetic data by taking interpolations of pairs of training data and has been shown to improve model generalizability (49, 50).”
• Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning (2021, methodology): “[70] Linjun Zhang, Zhun Deng, Kenji Kawaguchi, Amirata Ghorbani, and James Zou.”
• Shifting machine learning for healthcare from development to deployment and from models to data (2022, background): “There are also solutions for addressing data shifts that involve proactively addressing them during model development 175–178 or retroactively by surveilling for data shifts during model deployment 179 .”
• Advances, challenges and opportunities in creating data for trustworthy AI (2022, background): “Recent theoretical analyses demonstrate that data augmentation such as Mixup and pseudo-labelling smooths and regularizes AI models, thus improving model robustness and generalizability 77,78 .”
• Rethinking the Representational Continuity: Towards Unsupervised Continual Learning (2021, background): “The interpolated examples not only augments the past tasks’ instances in the replay buffer but also approximates a regularized loss minimization (Zhang et al., 2021).”
• Retrospective Adversarial Replay for Continual Learning (2022, methodology): “To improve the diversity and variation of the perturbed samples (thus increasing the learnable information from a limited-size buffer), we investigate the role of MixUp [58, 67, 68], a data augmentation technique applied together with RAR — we find that it brings substantial improvements when there ”