Linguistically-Informed Transformations (LIT): A Method for Automatically Generating Contrast Sets

By Chuanrong Li, Lin Shengshuo, Xinyi Wu, Xuhui Zhou, Shane Steinert-Threlkeld. 2020

44 citations recorded by the source corpus.

Although large-scale pretrained language models, such as BERT and RoBERTa, have achieved superhuman performance on in-distribution test sets, their performance suffers on out-of-distribution test sets (e.g., on contrast sets). Building contrast sets often requires human-expert annotation, which is expensive and hard to create on a large scale. In this work, we propose a Linguistically-Informed Transformation (LIT) method to automatically generate contrast sets, which enables practitioners to explore linguistic phenomena of interests as well as compose different phenomena. Experimenting with our method on SNLI and MNLI shows that current pretrained language models, although being claimed to contain sufficient linguistic knowledge, struggle on our automatically generated contrast sets. Furthermore, we improve models’ performance on the contrast sets by applying LIT to augment the training data, without affecting performance on the original data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Ana Marasovic
2. Matt Gardner
3. Sarah Wiegreffe
4. Matthew E. Peters
5. Noah A. Smith
6. Mohit Bansal
7. Tongshuang Sherry Wu
8. Hao Peng

In Computer Vision and Pattern Recognition

1. Matthew E. Peters
2. Ana Marasovic
3. Matt Gardner
4. Hao Peng
5. Alexis Ross
6. Tongshuang Sherry Wu
7. Sarah Wiegreffe
8. Yonatan Belinkov

In Natural Language Processing

1. Ana Marasovic
2. Matthew E. Peters
3. Matt Gardner
4. Sarah Wiegreffe
5. Tongshuang Sherry Wu
6. Hao Peng
7. Alexis Ross
8. Mohit Bansal

In Evaluation & Benchmarks

1. Ana Marasovic
2. Matthew E. Peters
3. Sarah Wiegreffe
4. Matt Gardner
5. Hao Peng
6. Alexis Ross
7. Tongshuang Sherry Wu
8. Noah A. Smith

Citations in the corpus

• Tailor: Generating and Perturbing Text with Semantic Controls (2021, methodology): “…related are the creation of minimally edited datasets, either through manual rewriting (Gard-ner et al., 2020; Kaushik et al., 2020), or creating perturbation templates (Andreas, 2020; Li et al., 2020; Ribeiro et al., 2020; Wu et al., 2019); T ailor reduces the human e ff orts these studies require.”
• Teach Me to Explain: A Review of Datasets for Explainable NLP (2021, methodology): “13 However, contrastive edits have been used to assess and improve robustness of NLP models (Kaushik et al., 2020; Gardner et al., 2020; Li et al., 2020) and might used for explainability too.”
• RICA: Evaluating Robust Inference Capabilities Based on Commonsense Axioms (2020, unknown): “Natural Language Inference NLI has been well studied and several benchmarks have been proposed (Bowman et al., 2015; Williams et al., 2018), which have a larger scope than just commonsense inference.”
• Explaining NLP Models via Minimal Contrastive Editing (MiCE) (2020, unknown): “13 Li et al. (2020a) generate counterfactuals using linguistically-informed transformations (LIT), and Yang et al. (2020) generate counterfactuals for binary financial text classifi-cation using grammatically plausible single-word edits (REP-SCD).”
• Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing (2021, methodology): “Outside of EXNLP, datasets with contrastive edits have been collected to assess and improve robustness of NLP models [59, 38, 74] and might be used for explainability too.”