Negative Data Augmentation
By Abhishek Sinha, Kumar Ayush, Jiaming Song, Burak Uzkent, Hongxia Jin, Stefano Ermon. 2021
83 citations recorded by the source corpus.
Data augmentation is often used to enlarge datasets with synthetic samples generated in accordance with the underlying data distribution. To enable a wider range of augmentations, we explore negative data augmentation strategies (NDA)that intentionally create out-of-distribution samples. We show that such negative out-of-distribution samples provide information on the support of the data distribution, and can be leveraged for generative modeling and representation learning. We introduce a new GAN training objective where we use NDA as an additional source of synthetic data for the discriminator. We prove that under suitable conditions, optimizing the resulting objective still recovers the true data distribution but can directly bias the generator towards avoiding samples that lack the desired structure. Empirically, models trained with our method achieve improved conditional/unconditional image generation along with improved anomaly detection capabilities. Further, we incorporate the same negative data augmentation strategy in a contrastive learning framework for self-supervised representation learning on images and videos, achieving improved performance on downstream image classification, object detection, and action recognition tasks. These results suggest that prior knowledge on what does not constitute valid data is an effective form of weak supervision across a range of unsupervised learning tasks.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Florian Tramèr
- Nicholas Carlini
- Eric Wallace
- Daphne Ippolito
- Matthew Jagielski
- Jamie Hayes
- Eduard H. Hovy
- Percy Liang
In Computer Vision and Pattern Recognition
- Florian Tramèr
- Nicholas Carlini
- Jamie Hayes
- Taghi M. Khoshgoftaar
- Eduard H. Hovy
- Connor Shorten
- Milad Nasr
- Borko Furht
In Representation & Self-Supervised Learning
- Nicholas Carlini
- Jason Wei
- Sarath Chandar
- Eduard H. Hovy
- Eric Wallace
- Connor Shorten
- Steven Y. Feng
- Teruko Mitamura
Citations in the corpus
- A Survey of Data Augmentation Approaches for NLP (2021, background): “ample, negative data augmentation (NDA) (Sinha et al., 2021) involves creating out-of-distribution samples.”
- Extracting Training Data from Diffusion Models (2023, methodology): “For this experiment, instead of using models we train ourselves (something that was necessary to run LiRA), we study five off-the-shelf pre-trained GANs: WGAN-ALP [68], E2GAN [69], NDA [63], DiffBigGAN [82], and StyleGAN-ADA [43].”
- Text Data Augmentation for Deep Learning (2021, background): “Sinha et al. [114] demonstrate how this can be used to improve contrastive learning and generative adversarial networks.”