Syntactic Data Augmentation Increases Robustness to Inference Heuristics

By Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, Tal Linzen. 2020

199 citations recorded by the source corpus.

Pretrained neural models such as BERT, when fine-tuned to perform natural language inference (NLI), often show high accuracy on standard datasets, but display a surprising lack of sensitivity to word order on controlled challenge sets. We hypothesize that this issue is not primarily caused by the pretrained model’s limitations, but rather by the paucity of crowdsourced NLI examples that might convey the importance of syntactic structure at the fine-tuning stage. We explore several methods to augment standard training sets with syntactically informative examples, generated by applying syntactic transformations to sentences from the MNLI corpus. The best-performing augmentation method, subject/object inversion, improved BERT’s accuracy on controlled examples that diagnose sensitivity to word order from 0.28 to 0.73, without affecting performance on the MNLI test set. This improvement generalized beyond the particular construction used for data augmentation, suggesting that augmentation causes BERT to recruit abstract syntactic representations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

How Can We Accelerate Progress Towards Human-like Linguistic Generalization? (2020, background): “importance of word order (Min et al., 2020).”
Tailor: Generating and Perturbing Text with Semantic Controls (2021, background): “Min et al. (2020) ﬁnd that augmenting SNLI training data by swapping hypotheses’ subject / objects ( e.g., This collection contains 16 El Grecos.”
An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Models (2020, background): “Recent work has shown that augmenting syntactically-rich examples improves robust accuracy on NLI (Min et al., 2020).”
Learning from others' mistakes: Avoiding dataset biases without modeling them (2020, methodology): “…and optimization efforts, data augmentation has attracted interest as a way to reduce model biases by explicitly modifying the dataset distribution (Min et al., 2020; Be-linkov & Bisk, 2018), either by leveraging human knowledge about dataset biases such as swapping male and female entities (Zhao…”
An Analysis of Simple Data Augmentation for Named Entity Recognition (2020, background): “…an increased interest on applying data augmentation techniques on sentence-level and sentence-pair natural language processing (NLP) tasks, such as text clas-siﬁcation (Wei and Zou, 2019; Xie et al., 2019), natural language inference (Min et al., 2020) and machine translation (Wang et al., 2018).”
Supervising Model Attention with Human Explanations for Robust Natural Language Inference (2021, methodology): “2020), using data-augmentation (Min et al. 2020; Minervini and Riedel 2018), fine-tuning on minority examples (Yaghoobzadeh et al.”
Learning Helpful Inductive Biases from Self-Supervised Pretraining (2020, background): “McCoy et al. (2019) and Min et al. (2020) show that small amounts of inoculating data during training on textual entailment help BERT overlook certain surface generalizations.”
Data Augmentation Approaches in Natural Language Processing: A Survey (2021, methodology): “Rules 最终版（右边这个小的） context ([54, 55]).”
Predicting Inductive Biases of Pre-Trained Models (2021, background): “Others use such datasets to improve models’ handling of linguistic features (Min et al., 2020; Poliak et al., 2018a; Liu et al., 2019a), or to mitigate biases (Zmigrod et al., 2019; Zhao et al., 2018; 2019; Hall Maudslay et al., 2019; Lu et al., 2020).”
Substructure Substitution: Structured Data Augmentation for NLP (2021, background): “…improve the robustness of neural network– based NLP models (Belinkov and Bisk, 2018; Shi et al., 2018a; Alzantot et al., 2018; Zhang et al., 2019; Min et al., 2020, inter alia ), or to generate counterfactual examples which mitigate bias in natural language (Zmigrod et al., 2019; Lu et al.,…”
Linguistically-Informed Transformations (LIT): A Method for Automatically Generating Contrast Sets (2020, methodology): “NLI Model Diagnosis Our work builds on works diagnosing and improving NLI models with automatically augmented instances (McCoy et al., 2019; Min et al., 2020).”
What Happens To BERT Embeddings During Fine-tuning? (2020, background): “…behavioral analyses have shown that finetuned models can still fail to leverage even simple syntactic knowledge in their predictions (McCoy et al., 2019b,a; Min et al., 2020), and may instead rely on annotation artifacts (Gururangan et al., 2018) or pattern matching (Jia and Liang, 2017).”
A Survey on Recent Approaches for Natural Language Processing in Low-Resource Scenarios (2020, background): “2548 of the subject-object relation (Min et al., 2020).”
Text Data Augmentation for Deep Learning (2021, background): “Min et al. [37] propose rules for augmentation based on syntactic heuristics.”
Understanding by Understanding Not: Modeling Negation in Language Models (2021, methodology): “Our method can be thought of a type data augmentation, which has be shown to be effective at improving robustness across many tasks in language, such as text classification (Wei and Zou, 2019), natural language inference (Min et al., 2020; McCoy et al., 2019) and semantic parsing (Andreas, 2019).”
Natural Language Inference with a Human Touch: Using Human Explanations to Guide Model Attention (2021, methodology): “, 2020), using data-augmentation (Min et al., 2020; Minervini and Riedel, 2018), fine-tuning on minority examples (Yaghoobzadeh et al.”