Trick Me If You Can: Adversarial Writing of Trivia Challenge Questions

By Eric Wallace, Pedro Rodriguez, Jordan L. Boyd-Graber. 2018

21 citations recorded by the source corpus.

Modern question answering systems have been touted as approaching human performance. However, existing question answering datasets are imperfect tests. Questions are written with humans in mind, not computers, and often do not properly expose model limitations. To address this, we develop an adversarial writing setting, where humans interact with trained models and try to break them. This annotation process yields a challenge set, which despite being easy for trivia players to answer, systematically stumps automated question answering systems. Diagnosing model errors on the evaluation data provides actionable insights to explore in developing robust and generalizable question answering systems.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, methodology): “We produced 437 questions this way, which we call the “filtered” questions (Wallace and Boyd-Graber, 2018).”
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, background): “The term has been used to refer both to examples purposefully generated to fool existing models [43,44], but also to hard natural examples that current models struggle with [9,13,71,79].”
White-to-Black: Efficient Distillation of Black-Box Adversarial Attacks (2019, background): “…include adding random or heuristically constructed noise (Belinkov and Bisk, 2017; Rodriguez and Rojas-Galeano, 2018; Gao et al., 2018), meaningpreserving modifications that change the surface form (Iyyer et al., 2018; Ribeiro et al., 2018), and human-in-the-loop generation (Wallace et al., 2018).”
Adversarial Attacks on Deep Learning Models in Natural Language Processing: A Survey (2019, methodology): “ness of feed forward network in NLP, researchers often works on speciﬁc architecture in real applications. For example, authors of [23], [24], [25] worked on the speciﬁed malware detection models and [26] was on question answering. 4.3.2 Convolutional Neural Network (CNN) Convolutional Neural Networ”