Know What You Don’t Know: Unanswerable Questions for SQuAD

By Pranav Rajpurkar, Robin Jia, Percy Liang. 2018

3,369 citations recorded by the source corpus.

Extractive reading comprehension systems can often locate the correct answer to a question in a context document, but they also tend to make unreliable guesses on questions for which the correct answer is not stated in the context. Existing datasets either focus exclusively on answerable questions, or use automatically generated unanswerable questions that are easy to identify. To address these weaknesses, we present SQuADRUn, a new dataset that combines the existing Stanford Question Answering Dataset (SQuAD) with over 50,000 unanswerable questions written adversarially by crowdworkers to look similar to answerable ones. To do well on SQuADRUn, systems must not only answer questions when possible, but also determine when no answer is supported by the paragraph and abstain from answering. SQuADRUn is a challenging natural language understanding task for existing models: a strong neural system that gets 86% F1 on SQuAD achieves only 66% F1 on SQuADRUn. We release SQuADRUn to the community as the successor to SQuAD.

Subfields: Artificial Intelligence, Information Systems and Management, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems and Management

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, background): “To remedy the deﬁciency of SQuAD, Rajpurkar, Jia, and Liang (2018) developed SQuAD 2.0 that combines SQuAD with new unanswerable questions.”
XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, background): “• For explicit reasoning tasks like SQuAD and RACE that involve longer context, the performance gain of XLNet is usually larger.”
PaLM: Scaling Language Modeling with Pathways (2022, background): “Examples: SQuADv2, Winograd.”
Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “Reading Comprehension In Table 23 we report zero-shot and few-shot results on SQUAD and zero-shot and one-shot experiments on QUAC.”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, background): “In the paper we sometimes abuse the term “language model” slightly to make the phrase shorter. such as SQuAD (Rajpurkar et al., 2016; 2018).”
UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “Among the datasets in this popular format, we adopt SQuAD 1.1 (Rajpurkar et al., 2016), SQuAD 2 (Rajpurkar et al., 2018), NewsQA (Trischler et al., 2017), Quoref (Dasigi et al., 2019), ROPES (Lin et al., 2019).”
Experience Grounds Language (2020, background): “Recent pretraining literature has produced results that few could have predicted, crowding leader boards re-sults that make claims to super-human accuracy (Rajpurkar et al., 2018).”
Adversarial Training for Large Neural Language Models (2020, background): “0 (Rajpurkar et al., 2016, 2018) are popular machine reading comprehension benchmarks.”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, methodology): “…models on three popular benchmarks: The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2018), two versions of the Stanford Question Answering Dataset (SQuAD; Rajpurkar et al., 2016; 2018), and the ReAding Comprehension from Examinations (RACE) dataset (Lai et al., 2017).”
TinyBERT: Distilling BERT for Natural Language Understanding (2019, neutral): “0 (Rajpurkar et al., 2018).”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “We ﬁnetune the trained models on several downstream tasks including MNLI and QQP from the GLUE benchmark (Wang et al., 2019), SQuAD 1.1 and SQuAD 2.0 from the Stanford Question answering dataset (Rajpurkar et al., 2016; 2018), and the reading comprehension RACE dataset (Lai et al., 2017).”
Probing Neural Network Comprehension of Natural Language Arguments (2019, background): “There is much recent work focusing on statistical cues in datasets in vision (Jo and Bengio, 2017) and NLP (Sanchez et al., 2018; McCoy et al., 2019; Gururangan et al., 2018; Glockner et al., 2018; Poliak et al., 2018; Rajpurkar et al., 2018; Jia and Liang, 2017).”
BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions (2019, methodology): “Following previous work (Hu et al., 2018), we also try building entailment-like training data from SQuAD 2.0 (Rajpurkar et al., 2018).”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “, 2018; Talmor and Berant, 2018), mismatched passages and questions (Saha et al., 2018; Kociský et al., 2018; Rajpurkar et al., 2018), integrating knowledge from external sources (Mihaylov et al.”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “An increasing number of analyses (Jia and Liang, 2017; Rajpurkar et al., 2018; Kaushik and Lipton, 2018) have revealed that a large portion of questions in these datasets can be answered by simply matching the patterns between the question and the answer sentence in the passage.”
CoQA: A Conversational Question Answering Challenge (2018, methodology): “not have any unanswerable questions, the later version (Rajpurkar et al., 2018) focuses solely on obtaining them, resulting in higher frequency than in CoQA.”