Trick Me If You Can: Human-in-the-Loop Generation of Adversarial Examples for Question Answering

By Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, Jordan L. Boyd-Graber. 2018

169 citations recorded by the source corpus.

Adversarial evaluation stress-tests a model’s understanding of natural language. Because past approaches expose superficial patterns, the resulting adversarial examples are limited in complexity and diversity. We propose human- in-the-loop adversarial generation, where human authors are guided to break models. We aid the authors with interpretations of model predictions through an interactive user interface. We apply this generation framework to a question answering task called Quizbowl, where trivia enthusiasts craft adversarial questions. The resulting questions are validated via live human–computer matches: Although the questions appear ordinary to humans, they systematically stump neural and information retrieval models. The adversarial questions cover diverse phenomena from multi-hop reasoning to entity type distractors, exposing open challenges in robust question answering.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “Wallace et al. (2019) leverage trivia experts to create a model-driven adversarial question writing procedure and generate a small set of challenge questions that QA-models fail on.”
Adversarial Training for High-Stakes Reliability (2022, methodology): “Like Wallace et al. [43], our tool displays a saliency map; however, we added a token substitution tool that gives humans automated suggestions to help hill-climb toward an adversarial example.”
Universal Adversarial Triggers for Attacking and Analyzing NLP (2019, background): “Other attack methods are based on generative (Iyyer et al., 2018) or human-in-the-loop approaches (Wallace et al., 2019).”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, methodology): “Adversarial examples are almost the methodological opposite of contrast sets: they change the input such that a model’s decision changes but the gold label does not (Jia and Liang, 2017; Wallace et al., 2019a).”
Pre-Trained Models: Past, Present and Future (2021, background): “Recently, human-in-the-loop methods (Wallace et al., 2019b; Nie et al., 2020) have been applied to generate more natural, valid, and diverse adversarial examples, which brings larger challenge and expose more properties and problems of PTMs.”
Evaluating NLP Models via Contrast Sets (2020, methodology): “Adversarial examples are almost the methodological opposite of contrast sets: they change the input such that a model’s decision changes but the gold label does not (Jia and Liang, 2017; Wallace et al., 2019a).”
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation (2021, methodology): “Other datasets for question answering (Rajpurkar et al., 2018; Dua et al., 2019; Wallace et al., 2019), sentiment analysis (Potts et al., 2021), hate speech detection (Vidgen et al., 2021), and natural language inference (Nie et al., 2020) have been collected in a similar manner.”
Misleading Failures of Partial-input Baselines (2019, methodology): “For example, models can be stress tested using adversarial examples (Jia and Liang, 2017; Wallace et al., 2019) and challenge sets (Glock-ner et al., 2018; Naik et al., 2018).”
What Question Answering can Learn from Trivia Nerds (2019, background): “One of the trivia experts consulted in Wallace et al. (2019) crafted a question that tripped up neural QA by embedding the phrase “this author opens Crime and Punishment” into a question; the top system confidently answers Fyodor Dostoyevski.”
On Evaluating Adversarial Robustness of Large Vision-Language Models (2023, background): “…observed in vision tasks, early efforts are devoted to investigating adversarial attacks against visual question answering [6, 11, 37, 38, 43, 77, 89, 97, 105] and image caption [1, 14, 99], with the majority of these efforts focusing on conventional CNN-RNN-based models, assuming white-box…”
Can You Unpack That? Learning to Rewrite Questions-in-Context (2019, background): “…(Sugawara et al., 2018), existing research either introduces new benchmarks of difﬁcult (adversarial) stand-alone questions (Dua et al., 2019; Wallace et al., 2019, inter alia), or models that simplify hard questions through paraphrasing (Dong et al., 2017) or decomposition (Talmor and…”
A Survey of Human-in-the-loop for Machine Learning (2021, methodology): “[56] developed an open application system that contains an interactive interface to talk with the machine, thereby generating more Q&A language materials to collect more research data and help the researcher explain the model predictions.”
Fool Me Twice: Entailment from Wikipedia Gamification (2021, methodology): “While this approach has been used for question answering (Wallace et al., 2019; Bartolo et al., 2020), which has a similar retrieval step, to our knowledge it has not been applied to entailment or FEVER .”
Universal Adversarial Triggers for NLP (2019, background): “Other attack methods are based on generative (Iyyer et al., 2018) or human-in-the-loop approaches (Wallace et al., 2019).”
On the Efficacy of Adversarial Data Collection for Question Answering: Results from a Large-Scale Randomized Study (2021, methodology): “Research on ADC has picked up recently, with Chen et al. (2019) tasking crowdworkers to construct multiple-choice questions to fool a BERT model and Wallace et al. (2019) employing Quizbowl community members to write Jeopardy-style questions to compete against QA models.”