Simple and Effective Multi-Paragraph Reading Comprehension

By Christopher Clark, Matt Gardner. 2017

485 citations recorded by the source corpus.

We introduce a method of adapting neural paragraph-level question answering models to the case where entire documents are given as input. Most current question answering models cannot scale to document or multi-document input, and naively applying these models to each paragraph independently often results in them being distracted by irrelevant text. We show that it is possible to significantly improve performance by using a modified training scheme that teaches the model to ignore non-answer containing paragraphs. Our method involves sampling multiple paragraphs from each document, and using an objective function that requires the model to produce globally correct output. We additionally identify and improve upon a number of other design decisions that arise when working with document-level data. Experiments on TriviaQA and SQuAD shows our method advances the state of the art, including a 10 point gain on TriviaQA.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, background): “Table 2 shows top leaderboard entries as well as results from top published systems (Seo et al., 2017; Clark and Gardner, 2018; Peters et al., 2018a; Hu et al., 2018).”
Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, background): “In spite of a variety of model structures and attention types (Cui et al., 2016; Xiong et al., 2016; Seo et al., 2016; Wang et al., 2017; Clark and Gardner, 2017), a typical attention-based neural network model for MRC ﬁrst encodes the symbolic representation of the question and passage in an…”
U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, methodology): “One approach is to directly extend previous MRC models by introducing a no-answer score to the score vector of the answer span (Levy et al. 2017; Clark and Gardner 2017).”
Deep Contextualized Word Representations (2018, methodology): “Our baseline model (Clark and Gardner, 2017) is an improved version of the Bidirectional Attention Flow model in Seo et al. (BiDAF; 2017).”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, neutral): “We compare our results to the popular extractive QA paradigm [5, 7, 27, 22], where answers are extracted as spans from retrieved documents, relying primarily on non-parametric knowledge.”
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, background): “Clark and Gardner (2018) proposed to use a global normalization over all the span corresponding to the answer, which was later applied to BERT based models (Wang et al., 2019).”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, methodology): “Following previous reading comprehension work (Rajpurkar et al., 2016; Seo et al., 2016; Lee et al., 2016; Clark & Gardner, 2017), we will assume that the answer y can be found as a contiguous sequence of tokens in some document z.”
Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “In open-domain QA, models can extract a span of text from retrieved documents as the answer (Chen et al., 2017; Clark & Gardner, 2018; Wang et al., 2019; Karpukhin et al., 2020), a method inspired by reading comprehension (Richardson, 2013; Rajpurkar et al., 2016).”
Pretrained Transformers Improve Out-of-Distribution Robustness (2020, methodology): “For reading comprehension, we use the DocQA model (Clark and Gardner, 2018) with GloVe embeddings.”
DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs (2019, methodology): “To address this, we adopted the marginal likelihood objective function proposed by Clark and Gardner (2018), which sums over the probabilities of all the matching spans.”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “DocQA 2 (Clark and Gardner, 2018) is a strong baseline model for queries with extractive answers.”
CoQA: A Conversational Question Answering Challenge (2018, methodology): “We anticipate that recent innovations such as self-attention (Clark and Gardner, 2018) and contextualized word embeddings (Peters et al.”
QuAC: Question Answering in Context (2018, background): “, 2016, BiDAF) with self-attention (Clark and Gardner, 2018) and contextualized embeddings.”
Know What You Don’t Know: Unanswerable Questions for SQuAD (2018, methodology): “Clark and Gardner (2017) generate negative examples for SQuAD by pairing existing questions with other paragraphs from the same article based on TF-IDF overlap; we refer to these as T F I DF examples.”
Big Bird: Transformers for Longer Sequences (2020, methodology): “Loss cross-entropy cross-entropy cross-entropy cross-entropy golden spans golden spans noisy spans [19] ans choices Compute resources 4× 2 TPUv3 4× 8 TPUv3 4× 2 TPUv3 4× 4 TPUv3”
CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge (2019, methodology): “DOCQA A state-of-the-art RC model (Clark and Gardner, 2017), that uses the retrieved Google web snippets (Section 3) as context.”
Augmented Language Models: a Survey (2023, methodology): “Various works augment LMs with a dense retriever by adding the retrieved documents to the current context (Chen et al., 2017; Clark and Gardner, 2017; Lee et al., 2019; Guu et al., 2020; Khandelwal et al., 2020; Lewis et al., 2020; Izacard and Grave, 2020; Zhong et al., 2022; Borgeaud et al., 2022;…”
Longformer: The Long-Document Transformer (2020, methodology): “Because of the distant supervision nature of the training data (no gold answer spans), we use the loss function of Clark and Gardner (2017) which works like an OR that the model only needs to get one answer span right, not all of them.”