Implicit Semantic Data Augmentation for Deep Networks

By Yulin Wang, Xuran Pan, Shiji Song, Hong Zhang, Cheng Wu, Gao Huang. 2019

242 citations recorded by the source corpus.

In this paper, we propose a novel implicit semantic data augmentation (ISDA) approach to complement traditional augmentation techniques like flipping, translation or rotation. Our work is motivated by the intriguing property that deep networks are surprisingly good at linearizing features, such that certain directions in the deep feature space correspond to meaningful semantic transformations, e.g., adding sunglasses or changing backgrounds. As a consequence, translating training samples along many semantic directions in the feature space can effectively augment the dataset to improve generalization. To implement this idea effectively and efficiently, we first perform an online estimate of the covariance matrix of deep features for each class, which captures the intra-class semantic variations. Then random vectors are drawn from a zero-mean normal distribution with the estimated covariance to augment the training data in that class. Importantly, instead of augmenting the samples explicitly, we can directly minimize an upper bound of the expected cross-entropy (CE) loss on the augmented training set, leading to a highly efficient algorithm. In fact, we show that the proposed ISDA amounts to minimizing a novel robust CE loss, which adds negligible extra computational cost to a normal training procedure. Although being simple, ISDA consistently improves the generalization performance of popular deep models (ResNets and DenseNets) on a variety of datasets, e.g., CIFAR-10, CIFAR-100 and ImageNet. Code for reproducing our results are available at https://github.com/blackfeather-wang/ISDA-for-Deep-Networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Gao Huang
2. Qi Tian
3. Lingxi Xie
4. Tao Mei
5. Shiji Song
6. Quoc V. Le
7. Serge J. Belongie
8. Alan L. Yuille

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Yi Yang
4. Alan L. Yuille
5. Qi Tian
6. Xiaogang Wang
7. Tao Mei
8. Trevor Darrell

In Representation & Self-Supervised Learning

1. Tao Mei
2. Ting Yao
3. Yingwei Pan
4. Serge J. Belongie
5. Ser-Nam Lim
6. Gao Huang
7. Qi Tian
8. Adrian Weller

In Optimization & Training

1. Serge J. Belongie
2. Kilian Q. Weinberger
3. Lingxi Xie
4. Gao Huang
5. Felix Wu
6. Shiji Song
7. Yulin Wang
8. Boyi Li

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, background): “Dynamic networks are compatible with most advanced techniques in deep learning, including architecture design [4], [5], optimization algorithms [23], [24] and data preprocessing [25], [26], which ensures that they can benefit from the most recent advances in the field to achieve state-of-the-art per”
• Continual Learning by Modeling Intra-Class Variation (2022, background): “The visualization framework proposed in ISDA is shown in Figure 11.”
• On Feature Normalization and Data Augmentation (2020, methodology): “[58] adjust either the features or loss function as implicit data augmentation methods.”
• Joint Contrastive Learning with Infinite Possibilities (2020, background): “However, both our bounding technique and the motivation fundamentally differ from ISDA.”
• Adaptive Focus for Efficient Video Recognition (2021, background): “The only difference is that TSM+ does not train the policy network π , since it adopts full frames as inputs.”
• Data Augmentation Revisited: Rethinking the Distribution Gap between Clean and Augmented Data (2019, methodology): “More recently, augmentation in the deep feature space by intra-class covariance matrices was efficiently implemented with a new loss function [42].”
• Glance and Focus: a Dynamic Approach to Reducing Spatial Redundancy in Image Classification (2020, methodology): “We adopt the same data augmentation and pre-processing configurations as [22, 14, 57].”