Trick Me If You Can: Human-in-the-loop Generation of Adversarial Question Answering Examples

By Eric Wallace, Pedro Rodriguez, Shi Feng, Ikuya Yamada, Jordan L. Boyd-Graber. 2019

28 citations recorded by the source corpus.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “Wallace et al. (2019) leverage trivia experts to create a model-driven adversarial question writing procedure and generate a small set of challenge questions that QA-models fail on.”
Evaluating Models’ Local Decision Boundaries via Contrast Sets (2020, methodology): “…are constructed using base-line models in the data collection process, either to ﬁlter out examples that existing models answer correctly (e.g., Dua et al., 2019; Dasigi et al., 2019) or to generate adversarial inputs (e.g., Zellers et al., 2018, 2019; Wallace et al., 2019b; Nie et al., 2019).”
Evaluating NLP Models via Contrast Sets (2020, methodology): “or to generate adversarial inputs (e.g., Zellers et al., 2018, 2019; Wallace et al., 2019b; Nie et al., 2019).”
Can You Unpack That? Learning to Rewrite Questions-in-Context (2019, background): “…(Sugawara et al., 2018), existing research either introduces new benchmarks of difﬁcult (adversarial) stand-alone questions (Dua et al., 2019; Wallace et al., 2019, inter alia), or models that simplify hard questions through paraphrasing (Dong et al., 2017) or decomposition (Talmor and…”
Automatic Construction of Evaluation Suites for Natural Language Generation Datasets (2021, methodology): “models perform poorly on [25, 26, 4] or crowd sourcing data that looks ordinary but is challenging 80 for models [27, 28, 29].”
Evaluation Examples are not Equally Informative: How should that change NLP Leaderboards? (2021, methodology): “Ideally, new data should challenge models through adversarial collection (Wallace et al., 2019b; Nie et al., 2020) and related methods (Gard-ner et al., 2020).”
Toward Deconfounding the Effect of Entity Demographics for Question Answering Accuracy (2021, methodology): “Finally, for QB, we use the competition winner from Wallace et al. (2019), a BERT-based reranker of a TF-IDF retriever.”
Human-Machine Collaboration Approaches to Build a Dialogue Dataset for Hate Speech Countering (2022, background): “Wallace et al. (2019) propose using model output to guide humans in the writing of adversarial examples for question-answering systems.”
What Question Answering can Learn from Trivia Nerds (2019, background): “One of the trivia experts consulted in Wallace et al. (2019) crafted a question that tripped up neural QA by embedding the phrase “this author opens Crime and Punishment” into a question; the top system confidently answers Fyodor Dostoyevski.”