ANLIzing the Adversarial Natural Language Inference Dataset

By Adina Williams, Tristan Thrush, Douwe Kiela. 2020

53 citations recorded by the source corpus.

We perform an in-depth error analysis of Adversarial NLI (ANLI), a recently introduced large-scale human-and-model-in-the-loop natural language inference dataset collected over multiple rounds. We propose a fine-grained annotation scheme of the different aspects of inference that are responsible for the gold classification labels, and use it to hand-code all three of the ANLI development sets. We use these annotations to answer a variety of interesting questions: which inference types are most common, which models have the highest performance on each reasoning type, and which types are the most challenging for state of-the-art models? We hope that our annotations will enable more fine-grained evaluation of models trained on ANLI, provide us with a deeper understanding of where models fail and succeed, and help us determine how to train better models in future.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Continual Few-Shot Learning for Text Classification (2021, methodology): “More precisely, we use the dataset with several linguistic categories annotated by Williams et al. (2020) from ANLI Round-3 (Nie et al., 2020); and also provide two new datasets with linguistic categories that we annotated using the coun-terfactual augmented data provided by Kaushik et al. (2020)…”
When Combating Hype, Proceed with Caution (2021, unknown): “This process produces difﬁcult test sets, and it can be a useful tool in understanding the limits of existing training sets or models (Williams et al., 2020).”
Adversarially Constructed Evaluation Sets Are More Challenging, but May Not Be Fair (2021, background): “Williams et al. (2020) further 172 provide fine-grained analysis of examples arising 173 from ANLI’s creation procedure.”
On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study (2021, methodology): “Williams et al. (2020) conducted an error analysis of model predictions on the datasets collected by Nie et al. (2020).”
SafeText: A Benchmark for Exploring Physical Safety in Language Models (2022, methodology): “For the NLI experiments, we utilize three RoBERTa-large models ﬁne-tuned on various NLI datasets: only MNLI (Williams et al., 2018), only SNLI (Bowman et al., 2015), and MNLI and SNLI with FEVER (Thorne et al., 2018), and ANLI (Williams et al., 2022).”