Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehension

By Max Bartolo, A. Roberts, Johannes Welbl, Sebastian Riedel, Pontus Stenetorp. 2020

200 citations recorded by the source corpus.

Abstract Innovations in annotation methodology have been a catalyst for Reading Comprehension (RC) datasets and models. One recent trend to challenge current RC models is to involve a model in the annotation process: Humans create questions adversarially, such that the model fails to answer them correctly. In this work we investigate this annotation methodology and apply it in three different settings, collecting a total of 36,000 samples with progressively stronger models in the annotation loop. This allows us to explore questions such as the reproducibility of the adversarial effect, transfer from data collected with varying model-in-the-loop strengths, and generalization to data collected without a model. We find that training on adversarially collected samples leads to strong generalization to non-adversarially collected datasets, yet with progressive performance deterioration with increasingly stronger models-in-the-loop. Furthermore, we find that stronger models can still learn from datasets collected with substantially weaker models-in-the-loop. When trained on data collected with a BiDAF model in the loop, RoBERTa achieves 39.9F1 on questions that it cannot answer when trained on SQuAD—only marginally lower than when trained on data collected using RoBERTa itself (41.0F1).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies (2021, methodology): “…set of an-notators to increase question diversity, limiting the number of questions a single annotator can write; and (c) we continuously train adversarial models during data collection, slowly increasing the difﬁ-culty in question writing and preventing recurring patterns (Bartolo et al., 2020).”
Avoiding catastrophic forgetting in mitigating model biases in sentence-pair classification with elastic weight consolidation (2020, background): “As RoBERTa (Liu et al., 2019b) has been shown to be more robust to adversarial training (Bartolo et al., 2020), we also include this in our evaluation.”
CommonsenseQA 2.0: Exposing the Limits of AI through Gamification (2021, methodology): “Examples for this include 368 [39, 40, 41, 12, 42, 34, 43, 38, 44], with recent works featuring re-training during dataset collection 369 (Beat-the-AI [22], StrategyQA [23] and Dynabench [21]).”
WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation (2022, background): “Beyond increasing annotator effort (Bartolo et al., 2020”
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, methodology): “1 Bartolo et al. (2020) and subsequent works ﬁnd that the performance degradation in the original evaluation setting when training on adversarially-collected data only is mitigated by also including some of the original training data.”
Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, methodology): “Table 10: An example task in the Answerability Classification category of our dataset, adopted from AdversarialQA (Bartolo et al., 2020).”
Dynabench: Rethinking Benchmarking in NLP (2021, background): “So far, there is evidence that adversarially collected data provides performance gains irrespective of the model in the loop (Nie et al., 2020; Dinan et al., 2019; Bartolo et al., 2020).”
QuALITY: Question Answering with Long Input Texts, Yes! (2021, methodology): “To our knowledge, this is a novel data collection method—it is inspired by adversarial collection methods (Nie et al., 2020; Bartolo et al., 2020) as a way of collecting more challenging data.”
Analyzing Dynamic Adversarial Training Data in the Limit (2021, methodology): “…133 versarial data collection has been widely adopted 134 in recent work, especially for building evaluation 135 datasets (Dua et al., 2019; Nie et al., 2020a; Di- 136 nan et al., 2019; Bartolo et al., 2020; Potts et al., 137 2021; Liu et al., 2021; Kaushik et al., 2021; Xu 138 et al., 2020, 2021).”
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation (2021, methodology): “It can also reduce the prevalence of dataset biases and annotator artefacts over time (Bartolo et al., 2020; Nie et al., 2020), since such phenomena can be subverted by model-fooling examples collected ar X iv :2 10 4.”
DynaSent: A Dynamic Benchmark for Sentiment Analysis (2020, background): “However, there are existing adversarial datasets that are large enough to support full-scale training efforts (Zellers et al., 2018, 2019; Chen et al., 2019; Dua et al., 2019; Bartolo et al., 2020).”
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, background): “Evaluating generalization overlaps with the growing interest in adversarial examples for evaluation [12, 16, 34, 36, 45, 49, 74].”
Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models (2021, background): “( iii ) Human-crafted adversarial examples are generally crafted in the human-in-the-loop manner [21, 38, 1] or use manually crafted templates to generate test cases [36, 43].”
Fool Me Twice: Entailment from Wikipedia Gamification (2021, methodology): “While this approach has been used for question answering (Wallace et al., 2019; Bartolo et al., 2020), which has a similar retrieval step, to our knowledge it has not been applied to entailment or FEVER .”
On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study (2021, methodology): “We ﬁne-tune three models (BERT, RoBERTa, and ELECTRA) on resulting datasets and evaluate them on held-out test sets, adversarial test sets from prior work (Bartolo et al., 2020), and 12 MRQA (Fisch et al., 2019) datasets.”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, unknown): “1 (Rajpurkar et al., 2016) and AdversarialQA (Bartolo et al., 2020)).”
A Survey of Human-in-the-loop for Machine Learning (2021, methodology): “One is the annotation of specific task datasets, such as entity extraction [50, 51], entity linking [59], and the other is more abstract tasks, such as Q&A tasks [56] and reading comprehension tasks [64].”