NewsQA: A Machine Comprehension Dataset

By Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, Kaheer Suleman. 2016

950 citations recorded by the source corpus.

We present NewsQA, a challenging machine comprehension dataset of over 100,000 human-generated question-answer pairs. Crowdworkers supply questions and answers based on a set of over 10,000 news articles from CNN, with answers consisting of spans of text in the articles. We collect this dataset through a four-stage process designed to solicit exploratory questions that require reasoning. Analysis confirms that NewsQA demands abilities beyond simple word matching and recognizing textual entailment. We measure human performance on the dataset and compare it to several strong neural models. The performance gap between humans and machines (13.3% F1) indicates that significant progress can be made on NewsQA through future research. The dataset is freely available online.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension (2017, methodology): “, 2016) 3 3 7 3 3 NewsQA(Trischler et al., 2016) 3 3 3 7* 7 WikiQA (Yang et al.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “, 2017] or NewsQA [Trischler et al., 2016]) to augment the small SQuAD training set.”
XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, background): “0 and NewsQA [31] for our leaderboard submission.”
UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “For completeness, we include the highest previous scores for each dataset; one must be careful when reading these numbers as the best previous numbers follow the fully supervised protocol (for NewsQA (Zhang et al., 2020), Quoref (Segal et al., 2019), DROP (Lan et al., 2019), ROPES (Lin et al., 2019)”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, methodology): “One primary type of questions these datasets lack are the ones that require reasoning over common sense or understanding across multiple sentences in the passage (Rajpurkar et al., 2016; Trischler et al., 2017).”
QuAC: Question Answering in Context (2018, methodology): “…2017) NarrativeQA (Kocisk´y et al., 2017) TriviaQA (Joshi et al., 2017) SQuAD 2.0 (Rajpurkar et al., 2018) MS Marco (Nguyen et al., 2016) NewsQA (Trischler et al., 2016) Table 1: Comparison of the QUAC dataset to other question answering datasets. tion”), which makes the dialogs more…”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, background): “…(Richardson et al., 2013), cloze-style (Hermann et al., 2015; Onishi et al., 2016; Hill et al., 2016), and span prediction (Rajpurkar et al., 2016; Trischler et al., 2017; Joshi et al., 2017) However, analysis (Chen et al., 2016; Sug-awara et al., 2017) of these datasets has shown that many of…”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “, 2017) and NewsQA (Trischler et al., 2017) could also represent question answering, SQuAD is particularly interesting because the human level performance of SQuAD models in the single-task setting depends on a quirk that does not generalize to all forms of question answering.”
Know What You Don’t Know: Unanswerable Questions for SQuAD (2018, background): “Machine reading comprehension has become a central task in natural language understanding, fueled by the creation of many large-scale datasets (Hermann et al., 2015; Hewlett et al., 2016; Rajpurkar et al., 2016; Nguyen et al., 2016; Trischler et al., 2017; Joshi et al., 2017).”
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (2018, background): “CNN/Daily Mail (Hermann et al., 2015), SQuAD (Rajpurkar et al., 2016) and NewsQA (Trischler et al., 2016) to name a few, are some of the datasets that were released recently with the goal of facilitating research in machine comprehension.”
Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge (2018, methodology): “Earlier reading comprehension datasets, e.g., MCTest (Richardson, 2013), SQuAD (Ra-jpurkar et al., 2016), NewsQA (Trischler et al., 2016), and CNN/DailyMail (Hermann et al., 2015), contained questions whose answers could be determined from surface-level cues alone (i.e., answers were “explicitly sta”
RACE: Large-scale ReAding Comprehension Dataset From Examinations (2017, methodology): “Articles of SQUAD, NEWSQA and MS MARCO come from Wikipedia, CNN news and the Bing search engine respectively.”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, methodology): “We train on 140 GB of text drawing from a wide variety of domains: Wikipedia (En, De, Es, Fr), Project Gutenberg 1 , submissions from 45 subreddits, OpenWebText 2 , a large collection of news data (Hermann et al., 2015; Barrault et al., 2019; Sandhaus, 2008; Grusky et al., 2018), Amazon Reviews (McA”
Dual Co-Matching Network for Multi-choice Reading Comprehension (2019, methodology): “, 2016) and NewsQA (Trischler et al., 2017) where the expected answer is usually in the form of a short span from the given passage, answer in multi-choice MRC is non-extractive and may not appear in the original passage, which allows rich types of questions such as commonsense reasoning and passage”
MLSUM: The Multilingual Summarization Corpus (2020, background): “Large scale datasets such as (Rajpurkar et al., 2016; Choi et al., 2018; Trischler et al., 2016) have driven fast progress.”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, methodology): “Question answering can also be interpreted as an exercise in verifying the knowledge of experts by finding the answer to trivia questions that are carefully crafted by someone who already knows the answer such that exactly one answer is correct such as TriviaQA and Quizbowl/Jeopoardy! questions (Fer”
ELI5: Long Form Question Answering (2019, methodology): “Extractive QA Extractive question answering datasets such as TREC (Voorhees, 2003), SQuAD (Rajpurkar et al., 2016, 2018), NewsQA (Trischler et al., 2017), SearchQA (Dunn et al., 2017), and QuAC (Choi et al., 2018) constrain the answer to a word or short phrase from the input and evaluate using exact”
SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine (2017, background): “Among these, examples with unigram answers are respectively 55,648, 8,672 and 17,056.”