Rethinking Data Augmentation: Self-Supervision and Self-Distillation

By Hankook Lee, Sung Ju Hwang, Jinwoo Shin. 2019

66 citations recorded by the source corpus.

Data augmentation techniques, e.g., flipping or cropping, which systematically enlarge the training dataset by explicitly generating more training samples, are effective in improving the generalization performance of deep neural networks. In the supervised setting, a common practice for data augmentation is to assign the same label to all augmented samples of the same source. However, if the augmentation results in large distributional discrepancy among them (e.g., rotations), forcing their label invariance may be too difficult to solve and often hurts the performance. To tackle this challenge, we suggest a simple yet effective idea of learning the joint distribution of the original and self-supervised labels of augmented samples. The joint learning framework is easier to train, and enables an aggregated inference combining the predictions from different augmented samples for improving the performance. Further, to speed up the aggregation process, we also propose a knowledge transfer technique, self-distillation, which transfers the knowledge of augmentation into the model itself. We demonstrate the effectiveness of our data augmentation framework on various fully-supervised settings including the few-shot and imbalanced classification scenarios.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Albumentations: fast and flexible image augmentations (2018, background): “However, it has also been reported that redundant or overly aggressive augmentation can hurt the performance and introduce biases into the dataset [36,37].”
Knowledge Distillation: A Survey (2020, methodology): “Furthermore, the knowledge transfer from one model to another in knowledge distillation can be extended to other tasks, such as adversarial attacks (Papernot et al. 2016), data augmentation (Lee et al. 2019a; Gordon and Duh 2019), data privacy and security (Wang et al. 2019a).”
Visual Commonsense R-CNN (2020, background): “They aim to learn visual features through an elaborated proxy task such as denoising autoencoders [6, 67], context & rotation prediction [13, 18] and data augmentation [33].”
CERT: Contrastive Self-supervised Learning for Language Understanding (2020, background): “, 2020a) with larger batch sizes and extensive data augmentation (Lee et al., 2019), which achieved results that are comparable with supervised learning.”
Data Augmentation in Classification and Segmentation: A Survey and New Strategies (2023, methodology): “For example, MNIST, CIFAR-10, CIFAR-100 and ImageNet are four commonly used datasets [10,33–35].”
Data Augmentation techniques in time series domain: a survey and taxonomy (2022, background): “In computer vision, it is quite popular to swap the RGB channels to perform DA[66].”
CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances (2020, methodology): “We observe that the classifying shifted instances loss L cls-SI (7) do not help supervised learning, which coincides with the observation of [34] that the self-supervised labels often conﬂict with the class labels.”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “The advancement of SSL [3], [4], [15], [16], [17], [18], [19], [20], [21], [22], [23], [24] has exhibitedrapidprogress, capturing signiﬁcant attention within the research community (Fig.”
Feature Space Augmentation for Long-Tailed Data (2020, methodology): “the state of the arts on addressing the long-tailed problem, including Classbalanced (CB) loss [9] based method and Focal Loss (FL) from [27] with various choices of hyper-parameters and augmentation based method [26].”