SSMBA: Self-Supervised Manifold Based Data Augmentation for Improving Out-of-Domain Robustness

By Nathan Ng, Kyunghyun Cho, Marzyeh Ghassemi. 2020

155 citations recorded by the source corpus.

Models that perform well on a training domain often fail to generalize to out-of-domain (OOD) examples. Data augmentation is a common method used to prevent overfitting and improve OOD generalization. However, in natural language, it is difficult to generate new examples that stay on the underlying data manifold. We introduce SSMBA, a data augmentation method for generating synthetic training examples by using a pair of corruption and reconstruction functions to move randomly on a data manifold. We investigate the use of SSMBA in the natural language domain, leveraging the manifold assumption to reconstruct corrupted text with masked language models. In experiments on robustness benchmarks across 3 tasks and 9 datasets, SSMBA consistently outperforms existing data augmentation methods and baseline models on both in-domain and OOD data, achieving gains of 0.8% accuracy on OOD Amazon reviews, 1.8% accuracy on OOD MNLI, and 1.4 BLEU on in-domain IWSLT14 German-English.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Douwe Kiela
3. Adina Williams
4. Eduard H. Hovy
5. Eric Michael Smith
6. Luke S. Zettlemoyer
7. Jason Aaron Edward Weston
8. Samuel R. Bowman

In Computer Vision and Pattern Recognition

1. Douwe Kiela
2. Mohit Bansal
3. Adina Williams
4. Eduard H. Hovy
5. Luc Van Gool
6. Dacheng Tao
7. Kyunghyun Cho
8. Trevor Darrell

In Representation & Self-Supervised Learning

1. Douwe Kiela
2. Adina Williams
3. Kyunghyun Cho
4. Eduard H. Hovy
5. Wanxiang Che
6. Yutai Hou
7. Bohan Li
8. Jason Wei

Citations in the corpus

• A Survey of Data Augmentation Approaches for NLP (2021, methodology): “Inspired by denoising autoencoders, Ng et al. (2020) use a corrupt-andreconstruct approach, with the corruption function q(x′|x) masking an arbitrary number of word positions and the reconstruction function r(x|x′) unmasking them using BERT (Devlin et al., 2019).”
• GPT3Mix: Leveraging Large-scale Language Models for Text Augmentation (2021, methodology): “Ng et al. (2020) proposed using masked language models as a denoising autoencoder to generate synthetic texts.”
• AAVAE: Augmentation-Augmented Variational Autoencoders (2021, methodology): “In natural language processing, Ng et al. [45] demonstrate the effectiveness of domain-specific augmentation by using a pretrained denoising autoencoder to build a robust classifier.”
• Perturbation Augmentation for Fairer NLP (2022, methodology): “Data augmentation has been shown to improve out of domain generalization (Ng et al., 2020; Wang et al., 2021a), and some work even use learned methods for doing data augmentation for syntactic alternatives (Ross et al.”
• Data Augmentation Approaches in Natural Language Processing: A Survey (2021, methodology): “[30] use the masked language model to construct a corruption model and a reconstruction model.”