On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networks

By Sunil Thulasidasan, Gopinath Chennupati, J. Bilmes, Tanmoy Bhattacharya, Sarah Ellen Michalak. 2019

628 citations recorded by the source corpus.

Mixup~\cite{zhang2017mixup} is a recently proposed method for training deep neural networks where additional samples are generated during training by convexly combining random pairs of images and their associated labels. While simple to implement, it has been shown to be a surprisingly effective method of data augmentation for image classification: DNNs trained with mixup show noticeable gains in classification performance on a number of image classification benchmarks. In this work, we discuss a hitherto untouched aspect of mixup training -- the calibration and predictive uncertainty of models trained with mixup. We find that DNNs trained with mixup are significantly better calibrated -- i.e., the predicted softmax scores are much better indicators of the actual likelihood of a correct prediction -- than DNNs trained in the regular fashion. We conduct experiments on a number of image classification architectures and datasets -- including large-scale datasets like ImageNet -- and find this to be the case. Additionally, we find that merely mixing features does not result in the same calibration benefit and that the label smoothing in mixup training plays a significant role in improving calibration. Finally, we also observe that mixup-trained DNNs are less prone to over-confident predictions on out-of-distribution and random-noise data. We conclude that the typical overconfidence seen in neural networks, even on in-distribution data is likely a consequence of training with hard labels, suggesting that mixup be employed for classification tasks where predictive uncertainty is a significant concern.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion (2021, background): “MixUp was shown to have two main effects: (1) it diversifies the training images and thus enlarges the training distribution on the vicinity of each training sample [7] and (2) it improves the network calibration [28, 68], reducing the overconfidence in recent classes.”
Essentials for Class Incremental Learning (2021, methodology): “Mixup has successfully been used as a form of data augmentation in image classification, improving generalization and calibration [32, 26].”
A survey of uncertainty in deep neural networks (2021, methodology): “[278] regularized it by using a data-agnostic data augmentation technique named”
Generalized Out-of-Distribution Detection: A Survey (2021, positive): “Similarly, other simple and effective methods to enhance model uncertainty estimation such as Ensemble (Dietterich, 2000) and Mixup (Thulasidasan et al., 2019) also demonstrate excellent performance.”
Decoupled Mixup for Data-efficient Learning (2022, positive): “DNNs trained with this technique are typically more generalizable and calibrated [39], whose predictions tend to be less overconfident.”
Remix: Rebalanced Mixup (2020, background): “Mixup also shed lights upon other learning tasks such as semi-supervised learning [33,1], adversarial defense [31] and neural network calibration [30].”
Retrospective Adversarial Replay for Continual Learning (2022, methodology): “…and variation of the perturbed samples (thus increasing the learnable information from a limited-size buffer), we investigate the role of MixUp [58, 67, 68], a data augmentation technique applied together with RAR — we ﬁnd that it brings substantial improvements when there are strict buffer…”
i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning (2021, background): “Not only improving the generalization on the supervised task, it also improves adversarial robustness (Pang et al., 2019) and conﬁdence calibration (Thulasidasan et al., 2019).”
A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “[236] investigated the predictive uncertainty and calibration of models trained with”
Energy-Based Open-World Uncertainty Modeling for Confidence Calibration (2021, methodology): “In particular, label smoothing modiﬁes the ground-truth labels by fusing them with a uniform distribution, essentially forcing neural networks to produce ‘more ﬂattened’ probabilities; whereas Mixup is a data augmentation method that randomly mixes two instances at both the image and label space, wi”
Isotropic Maximization Loss and Entropic Score: Fast, Accurate, Scalable, Unexposed, Turnkey, and Native Neural Networks Out-of-Distribution Detection. (2019, neutral): “However, we speculate this could also be achieved in a better way using isotropic regularization or special data augmentation techniques (Thulasidasan et al., 2019; Yun et al., 2019) to avoid the need for out-of-distribution or adversarial samples.”
Revisiting the Calibration of Modern Neural Networks (2021, methodology): “…modern neural networks to be poorly calibrated, as suggested previously (Guo et al., 2017; Lakshminarayanan et al., 2017; Malinin & Gales, 2018; Thulasidasan et al., 2019; Hendrycks et al., 2020b; Ovadia et al., 2019; Wenzel et al., 2020; Havasi et al., 2021; Rahaman & Thiery, 2020; Leathart &…”
How Does Mixup Help With Robustness and Generalization? (2020, background): “It has been empirically shown to substantially improve test performance and robustness to adversarial noise of state-of-the-art neural network architectures (Zhang et al., 2018; Lamb et al., 2019; Thulasidasan et al., 2019; Zhang et al., 2018; Arazo et al., 2019).”
Distinction Maximization Loss: Fast, Scalable, Turnkey, and Native Neural Networks Out-of-Distribution Detection simply by Replacing the SoftMax Loss (2019, methodology): “uts to decide works even better when many classes are presented. We speculate that recent advances in data augmentation techniques may help to improve IsoMax+ES OOD detection performance even further [52], [53]. 4.3 Robustness Analysis Fig. 3 presents OOD detection performance of SoftMax and IsoMax ”
In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning (2021, methodology): “Since the output prediction with mixup augmentation is better calibrated (Thulasidasan et al., 2019) we use relaxed thresholds for τ p ( 0 .”
Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learning (2019, methodology): “(7) As shown in [26], overconfidence in deep neural networks is a consequence of training on hard labels and it is the label smoothing effect from randomly combining yp and yq during mixup training that reduces prediction confidence and improves”
Bayes-Probe: Distribution-Guided Sampling for Prediction Level Sets (2020, result): “While many approaches aim to address this neural network overconfidence problem (e.g. Thulasidasan et al., 2019; Lee et al., 2017; Gal & Ghahramani, 2016; Blundell et al., 2015), our work is complementary to these efforts.”
MixCo: Mix-up Contrastive Learning for Visual Representation (2020, methodology): “In this paper, inspired by the success of mix-up training [34] in recent studies [29, 30, 32], we propose Mix-up Contrast (MixCo) as an extension of contrastive learning approach.”
Towards Out-Of-Distribution Generalization: A Survey (2021, background): “Unlike OOD generalization tasks, which primarily focus on performance under distribution shifts, the OOD detection community [211, 152, 98, 244] concentrates more on detecting samples from unseen distributions.”