SQuAD: 100,000+ Questions for Machine Comprehension of Text
By Pranav Rajpurkar, Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, Percy Liang. 2016
9,586 citations recorded by the source corpus.
We present the Stanford Question Answering Dataset (SQuAD), a new reading comprehension dataset consisting of 100,000+ questions posed by crowdworkers on a set of Wikipedia articles, where the answer to each question is a segment of text from the corresponding reading passage. We analyze the dataset to understand the types of reasoning required to answer the questions, leaning heavily on dependency and constituency trees. We build a strong logistic regression model, which achieves an F1 score of 51.0%, a significant improvement over a simple baseline (20%). However, human performance (86.8%) is much higher, indicating that the dataset presents a good challenge problem for future research. The dataset is freely available at this https URL
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Yejin Choi
- Graham Neubig
- Noah A. Smith
- Caiming Xiong
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Mohit Bansal
In Natural Language Processing
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Yejin Choi
- Graham Neubig
- Percy Liang
- Hannaneh Hajishirzi
- Noah A. Smith
In Evaluation & Benchmarks
- Mohit Bansal
- Yejin Choi
- Luke S. Zettlemoyer
- Hannaneh Hajishirzi
- Samuel R. Bowman
- Douwe Kiela
- Jianfeng Gao
- Ashish Sabharwal
Citations in the corpus
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “…task-specific architectures, ELMo advances the state-of-the-art for several major NLP benchmarks (Peters et al., 2018) including question answering (Rajpurkar et al., 2016) on SQuAD, sentiment analysis (Socher et al., 2013), and named entity recognition (Tjong Kim Sang and De Meulder, 2003).”
- Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, background): “AddSent AddOneSent Logistic (Rajpurkar et al., 2016) 23.2 30.4 Match-S (Wang and Jiang, 2016) 27.3 39.0 Match-E (Wang and Jiang, 2016) 29.4 41.8 BiDAF-S (Seo et al., 2016) 34.3 45.7 BiDAF-E (Seo et al., 2016) 34.2 46.9 ReasoNet-S (Shen et al., 2017) 39.4 50.3 ReasoNet-E (Shen et al., 2017) 39.4…”
- U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, methodology): “The best systems have even surpassed human performance on the Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al. 2016), one of the most widely used MRC benchmarks.”
- GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “QNLI The Stanford Question Answering Dataset (Rajpurkar et al. 2016) is a questionanswering dataset consisting of questionparagraph pairs, where one of the sentences in the”
- Deep Contextualized Word Representations (2018, background): “Question answering The Stanford Question Answering Dataset (SQuAD) (Rajpurkar et al., 2016) contains 100K+ crowd sourced question-answer pairs where the answer is a span in a given Wikipedia paragraph.”
- QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, methodology): “Their popularity can be attributed to an increase in publicly available annotated datasets, such as SQuAD (Rajpurkar et al., 2016), TriviaQA (Joshi et al., 2017), CNN/Daily News (Hermann et al., 2015), WikiReading (Hewlett et al., 2016), Children Book Test (Hill et al., 2015), etc.”
- Simple and Effective Multi-Paragraph Reading Comprehension (2017, methodology): “…of the questions; TriviaQA web, a dataset derived from TriviaQA unfiltered by treating each question-document pair where the document contains the question answer as an individual training point; and SQuAD (Rajpurkar et al., 2016), a collection of Wikipedia articles and crowdsourced questions.”
- TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension (2017, methodology): “, 2013), SQuAD (Rajpurkar et al., 2016), and NewsQA (Trischler”
- Bidirectional Attention Flow for Machine Comprehension (2016, methodology): “We evaluate the performance of our comprehension system on both SQuAD and CNN/DailyMail datasets.”
- Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “SQuAD [Rajpurkar et al., 2016] is a common question-answering benchmark.”
- RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, neutral): “0 (Rajpurkar et al., 2016, 2018).”
- XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, background): “79 Test set results on leaderboard, with data augmentation (as of June 19, 2019) Human [27] 82.”
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, methodology): “1 (Rajpurkar et al., 2016) is a reading comprehension dataset.”
- REALM: Retrieval-Augmented Language Model Pre-Training (2020, methodology): “In the paper we sometimes abuse the term “language model” slightly to make the phrase shorter. such as SQuAD (Rajpurkar et al., 2016; 2018).”
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “In open-domain QA, models can extract a span of text from retrieved documents as the answer (Chen et al., 2017; Clark & Gardner, 2018; Wang et al., 2019; Karpukhin et al., 2020), a method inspired by reading comprehension (Richardson, 2013; Rajpurkar et al., 2016).”
- Adversarial Training for Large Neural Language Models (2020, background): “0 (Rajpurkar et al., 2016, 2018) are popular machine reading comprehension benchmarks.”
- How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, background): “The most popular variant of this task feeds the model some “context” containing the answer (for example, a paragraph from an encyclo-pedia article) alongside the question (Rajpurkar et al., 2016; Zhang et al., 2018; Khashabi et al., 2018; Clark et al., 2019).”