How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?

By Shayne Longpre, Yu Wang, Christopher DuBois. 2020

94 citations recorded by the source corpus.

Task-agnostic forms of data augmentation have proven widely effective in computer vision, even on pretrained models. In NLP similar results are reported most commonly for low data regimes, non-pretrained models, or situationally for pretrained models. In this paper we ask how effective these techniques really are when applied to pretrained transformers. Using two popular varieties of task-agnostic data augmentation (not tailored to any particular task), Easy Data Augmentation (Wei andZou, 2019) and Back-Translation (Sennrichet al., 2015), we conduct a systematic examination of their effects across 5 classification tasks, 6 datasets, and 3 variants of modern pretrained transformers, including BERT, XLNet, and RoBERTa. We observe a negative result, finding that techniques which previously reported strong improvements for non-pretrained models fail to consistently improve performance for pretrained transformers, even when training data is limited. We hope this empirical analysis helps inform practitioners where data augmentation techniques may confer improvements.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Quoc V. Le
3. Luke S. Zettlemoyer
4. Yi Tay
5. Percy Liang
6. Noah A. Smith
7. Graham Neubig
8. Eduard H. Hovy

In Computer Vision and Pattern Recognition

1. Quoc V. Le
2. Luc Van Gool
3. Dacheng Tao
4. Trevor Darrell
5. Mohit Bansal
6. Xiaogang Wang
7. Li Fei-Fei
8. Ming-Hsuan Yang

In Evaluation & Benchmarks

1. Tongshuang Sherry Wu
2. Yi Tay
3. Barret Zoph
4. Quoc V. Le
5. Daniel S. Weld
6. Denny Zhou
7. Adam Roberts
8. Jason Wei

Citations in the corpus

• The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “These benefits invigorate the prospect of data augmentation techniques for LLM finetuning, which had previously been shown to have diminishing returns the longer models are pretrained (Longpre et al., 2020).”
• A Survey of Data Augmentation Approaches for NLP (2021, background): “…to light that a couple previously effective DA techniques for certain English text classification tasks (Wei and Zou, 2019; Sennrich et al., 2016) provide little ben-efit for models like BERT and RoBERTa, which already achieve high performance on in-domain text classification (Longpre et al., 2020).”
• Polyjuice: Generating Counterfactuals for Explaining, Evaluating, and Improving Models (2021, background): “We suspect that Polyjuice lacks domain knowledge and context for identifying critical perturbations, and therefore brings benefits redundant with pretraining (Longpre et al., 2020).”
• Text Data Augmentation for Deep Learning (2021, background): “Exploring back-translation augmentation for question answering Longpre et al. discuss “curating our input data and learning regime to encourage representations that are not biased by any one domain or distribution” [56].”
• A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “Recently, Longpre et al. (2020) hypothesised that data augmentation provides the same benefits as pretraining in transformer models.”
• A Survey on Data Augmentation for Text Classification (2021, unknown): “The described performance differences become apparent when approaching the challenge highlighted by Longpre et al. [4] from an intuitive perspective.”