Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup

By Jang-Hyun Kim, Wonho Choo, Hyun Oh Song. 2020

471 citations recorded by the source corpus.

While deep neural networks achieve great performance on fitting the training distribution, the learned networks are prone to overfitting and are susceptible to adversarial attacks. In this regard, a number of mixup based augmentation methods have been recently proposed. However, these approaches mainly focus on creating previously unseen virtual examples and can sometimes provide misleading supervisory signal to the network. To this end, we propose Puzzle Mix, a mixup method for explicitly utilizing the saliency information and the underlying statistics of the natural examples. This leads to an interesting optimization problem alternating between the multi-label objective for optimal mixing mask and saliency discounted optimal transport objective. Our experiments show Puzzle Mix achieves the state of the art generalization and the adversarial robustness results compared to other mixup methods on CIFAR-100, Tiny-ImageNet, and ImageNet datasets. The source code is available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

SSUL: Semantic Segmentation with Unknown Label for Exemplar-based Class-Incremental Learning (2021, background): “For example, the weakly-supervised semantic segmentation approaches [29, 31, 14] generate pseudo-labels filtering out the background regions using saliency map, and recent data augmentations [17, 16] also exploit the saliency information to find the optimal mixing of mask.”
Rainbow Memory: Continual Learning with a Memory of Diverse Samples (2021, methodology): “In particular, label mixing-based data augmentation, such as Cut-Mix [49], projects the input samples into a more complex dimension by mixing the image-label of multiple data samples randomly and has reported notable successes in various recognition tasks [47, 26].”
ResizeMix: Mixing Data with Preserved Object Information and True Labels (2020, methodology): “, PuzzleMix [29] proposes to optimize the mixing mask and the saliency discounted transportation, and SaliencyMix [50] uses a saliency detection module to select the saliency source patch for mixing.”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “[64] Jang-Hyun Kim, Wonho Choo, and Hyun Oh Song.”
Co-Mixup: Saliency Guided Joint Mixup with Supermodular Diversity (2021, methodology): “With this setup, we train a vanilla classiﬁer and reproduce the mixup baselines (Zhang et al., 2018; Verma et al., 2019; Yun et al., 2019; Kim et al., 2020), which we denote as Vanilla, Input, Manifold, CutMix, Puzzle Mix in the experiment tables.”
A Comprehensive Survey of Image Augmentation Techniques for Deep Learning (2022, background): “To account for the most important area, PuzzleMix [32] discriminates the foreground from the background, and mixes important information within the foreground.”
How Does Mixup Help With Robustness and Generalization? (2020, methodology): “This has motivated a recent line of work for developing a number of variants of Mixup, including Manifold Mixup (Verma et al., 2019a), Puzzle Mix (Kim et al., 2020), CutMix (Yun et al., 2019), Adversarial Mixup Resynthesis (Beckham et al., 2019), and PatchUp (Faramarzi et al., 2020).”
MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, background): “Several works [16, 17] have found that neural networks tend to rely upon texture information for making predictions, which consequently makes them vulnerable to out-of-distribution samples.”
Interpretable deep learning: interpretation, interpretability, trustworthiness, and beyond (2021, methodology): “Many weakly object localization and weakly semantic segmentation methods [7, 89, 182] start from an interpretation, and obtain promising results.”