The NarrativeQA Reading Comprehension Challenge

By Tomás Kociský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, Edward Grefenstette. 2017

1,062 citations recorded by the source corpus.

Reading comprehension (RC)—in contrast to information retrieval—requires integrating information and reasoning about events, entities, and their relations across a full document. Question answering is conventionally used to assess RC ability, in both artificial agents and children learning to read. However, existing RC datasets and tasks are dominated by questions that can be solved by selecting answers using superficial information (e.g., local context similarity or global term frequency); they thus fail to test for the essential integrative aspect of RC. To encourage progress on deeper comprehension of language, we present a new dataset and set of tasks in which the reader must answer questions about stories by reading entire books or movie scripts. These tasks are designed so that successfully answering their questions requires understanding the underlying narrative rather than relying on shallow pattern matching or salience. We show that although humans solve the tasks easily, standard RC models struggle on the tasks presented here. We provide an analysis of the dataset and the challenges it presents.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “The datasets used from this format are: NarrativeQA/NarQA (Kociský et al., 2018), the open-domain version of Natu-ralQuestions/NatQA (Kwiatkowski et al., 2019), and DROP (Dua et al., 2019b).”
Natural Questions: A Benchmark for Question Answering Research (2019, background): “, 2018), NarrativeQA (Kocisky et al., 2018), and HotpotQA (Yang et al.”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “There are other large-scale datasets (Nguyen et al., 2016; Joshi et al., 2017; Lai et al., 2017; Dunn et al., 2017; Kocisky et al., 2018; Reddy et al., 2018; Choi et al., 2018; Yang et al., 2018) targeting different aspects of reading comprehension.”
CoQA: A Conversational Question Answering Challenge (2018, methodology): “, 2017) 7 Multiple choice Mid/High School Exams Narrative QA (Kočiskỳ et al., 2018) 7 Free-form text Movie Scripts, Literature SQuAD 2.”
QuAC: Question Answering in Context (2018, background): “CoQA (Reddy et al., 2018) CSQA (Saha et al., 2018) CQA (Talmor and Berant, 2018) SQA (Iyyer et al., 2017) NarrativeQA (Kocisk´y et al., 2017) TriviaQA (Joshi et al., 2017) SQuAD 2.0 (Rajpurkar et al., 2018) MS Marco (Nguyen et al., 2016) NewsQA (Trischler et al., 2016) Table 1: Comparison of the QUA”
Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering (2018, background): “NarrativeQA (Kociský et al., 2017) is an RC dataset that has been shown to require an iterative reasoning about the narrative of a story.”
Compressive Transformers for Long-Range Sequence Modelling (2019, methodology): “The NarrativeQA Book Comprehension Task (Kočiskỳ et al., 2018) uses Project Gutenberg texts paired with Wikipedia articles, which can be used as summaries.”
The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, neutral): “NarrativeQA (Kočiskỳ et al., 2018) 19.”
SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, background): “The summarization datasets are trained for 10 epochs, while Qasper, QuALITY, and ContractNLI are trained for 20; NarrativeQA (the largest dataset) is trained for 2 epochs.”
ELI5: Long Form Question Answering (2019, methodology): “Abstractive QA Abstractive datasets include NarrativeQA (Kocisky et al., 2018), a dataset of movie and book summaries and CoQA (Reddy et al.”
Recursively Summarizing Books with Human Feedback (2021, methodology): “Then, we trained RL policies starting from each of the ¼ and ½ BC policies, 10 with approximately the same ROUGE-L BLEU-1 BLEU-4 METEOR BiDAF (Koˇcisk`y et al., 2018) 6.2 5.7 0.3 3.7 BM25 + BERT (Mou et al., 2020) 15.5 14.5 1.4 5.0 RoBERTa (Zemlyanskiy et al., 2021) 18.0 18.0 2.6 5.4 ETC (Zemlyanski”
Is Reinforcement Learning (Not) for Natural Language Processing?: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization (2022, background): “NarrativeQA (Kočiskỳ et al., 2018) deals with task of generating answers to questions about a given story.”