A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task

By Danqi Chen, Jason Bolton, Christopher D. Manning. 2016

597 citations recorded by the source corpus.

Enabling a computer to understand a document so that it can answer comprehension questions is a central, yet unsolved goal of NLP. A key factor impeding its solution by machine learned systems is the limited availability of human-annotated data. Hermann et al. (2015) seek to solve this problem by creating over a million training examples by pairing CNN and Daily Mail news articles with their summarized bullet points, and show that a neural network can then be trained to give good performance on this task. In this paper, we conduct a thorough examination of this new reading comprehension task. Our primary aim is to understand what depth of language understanding is required to do well on this task. We approach this from one side by doing a careful hand-analysis of a small subset of the problems and from the other by showing that simple, carefully designed systems can obtain accuracies of 73.6% and 76.6% on these two datasets, exceeding current state-of-the-art results by 7-10% and approaching what we believe is the ceiling for performance on this task.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, methodology): “Chen et al. devise a simple and effective document reader, by introducing a bilinear match function and a few manual features (Chen et al., 2017a).”
TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension (2017, methodology): “Finally, we present baseline experiments on the TriviaQA dataset, including a linear classifier inspired by work on CNN Dailymail and MCTest (Chen et al., 2016; Richardson et al., 2013) and a state-of-the-art neural network baseline (Seo et al., 2017).”
Bidirectional Attention Flow for Machine Comprehension (2016, methodology): “Chen et al. (2016) show that simply using bilinear term for computing the attention weights in the same model drastically improves the accuracy.”
SQuAD: 100,000+ Questions for Machine Comprehension of Text (2016, background): “While the size of this dataset is impressive, Chen et al. (2016) showed that the dataset requires less reasoning than previously thought, and concluded that performance is almost saturated.”
NumNet: Machine Reading Comprehension with Numerical Reasoning (2019, neutral): “Some works (Chen et al., 2016; Sugawara et al., 2018; Kaushik and Lipton, 2018) classify the reasoning skills required to answer the questions into the following types: (1) Exact matching/Paraphrasing; (2) Summary; (3) Logic reasoning; (4) Utilizing external knowledge; (5) Numerical reasoning.”
Natural Questions: A Benchmark for Question Answering Research (2019, background): “We also believe that, because a solution to NQ will have genuine utility, it is better equipped as a benchmark for NLU.”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, background): “…comprehension , where systems must understand a single passage of text well enough to answer arbitrary questions about it, has seen signiﬁcant progress in the last few years, so much that the most popular datasets available for this task have been solved (Chen et al., 2016; De-vlin et al., 2018).”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “, 2016) are closely related to ReCoRD: (1) The CNN/Daily Mail Corpus constructed queries from the bullet points, most of which required limited reasoning ability (Chen et al., 2016).”
CoQA: A Conversational Question Answering Challenge (2018, methodology): “As is well known, state-of-the-art models rely heavily on lexical similarity between a question and a passage (Chen et al., 2016; Weissenborn et al., 2017).”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, background): “…Onishi et al., 2016; Hill et al., 2016), and span prediction (Rajpurkar et al., 2016; Trischler et al., 2017; Joshi et al., 2017) However, analysis (Chen et al., 2016; Sug-awara et al., 2017) of these datasets has shown that many of the questions can be solved with context token matching (Chen et…”
Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences (2018, background): “Despite all the excitement fueled by that large data sets and the ability to directly train statistical learning models, current QA systems do not have capabilities comparable to elementary school or younger children (Clark and Etzioni, 2016).”
RACE: Large-scale ReAding Comprehension Dataset From Examinations (2017, methodology): “However, both require limited reasoning ability (Chen et al., 2016).”
NewsQA: A Machine Comprehension Dataset (2016, methodology): “ES The forms of reasoning required to solve NewsQA directly inﬂuence the abilities that models will learn from the dataset. We stratiﬁed reasoning types using a variation on the taxonomy presented by Chen et al. (2016) in their analysis of the CNN/Daily Mail dataset. Types are as follows, in ascendi”
Option Comparison Network for Multiple-choice Reading Comprehension (2019, methodology): “With the rapid development of deep learning, various neural models have been proposed for MCRC and achieve promising re-sults in recent years (Chen et al., 2016; Yin et al., 2016; Trischler et al., 2016; Dhingra et al., 2017; Tay et al., 2018; Parikh et al., 2018; Zhu et al., 2018; Wang et al.,…”
Improving Question Answering with External Knowledge (2019, background): “…Rajpurkar et al., 2016; Lai et al., 2017; Mihaylov et al., 2018; Clark et al., 2018; Choi et al., 2018; Reddy et al., 2018; Sun et al., 2019) and models (Chen et al., 2016; Wang et al., 2018b; Radford et al., 2018; Devlin et al., 2018; Sun et al., 2018) to drive progress in question answering.”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Although potentially any amount of reasoning is required to propose a response, many such tasks end up with sophisticated word overlap methods providing strong baselines (Chen, Bolton, and Manning, 2016).”
Reading Wikipedia to Answer Open-Domain Questions (2017, methodology): “Our Document Reader model is inspired by the recent success of neural network models on machine comprehension tasks, in a similar spirit to the At-tentiveReader described in (Hermann et al., 2015; Chen et al., 2016).”
Coarse-to-Fine Question Answering for Long Documents (2016, background): “State-of-the-art question answering models (Chen et al., 2016; Seo et al., 2016) use RNN models to encode the document and question and selects the answer.”
Denoising Distantly Supervised Open-Domain Question Answering (2018, methodology): “Recently, with the development of machine reading comprehension technique (Chen et al., 2016; Dhingra et al., 2017a; Cui et al., 2017; Shen et al., 2017; Wang et al., 2017), researchers attempt to answer open-domain questions via performing reading comprehension on plain texts.”