Learning Recurrent Span Representations for Extractive Question Answering

By Kenton C. T. Lee, Tom Kwiatkowski, Ankur P. Parikh, Dipanjan Das. 2016

154 citations recorded by the source corpus.

The reading comprehension task, that asks questions about a given evidence document, is a central problem in natural language understanding. Recent formulations of this task have typically focused on answer selection from a set of candidates pre-defined manually or through the use of an external NLP pipeline. However, Rajpurkar et al. (2016) recently released the SQUAD dataset in which the answers can be arbitrary strings from the supplied text. In this paper, we focus on this answer extraction task, presenting a novel model architecture that efficiently builds fixed length representations of all spans in the evidence document with a recurrent network. We show that scoring explicit span representations significantly improves performance over other approaches that factor the prediction into separate predictions about words or start and end markers. Our approach improves upon the best published results of Wang & Jiang (2016) by 5% and decreases the error of Rajpurkar et al.’s baseline by > 50%.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, methodology): “We use 3 binary features: exact match, lower-case match and lemma match between the question and passage (Lee et al. 2016).”
QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, neutral): “…et al., 2017) 68.4 / 77.1 68.4 / 77.1 BiDAF (Seo et al., 2016) 68.0 / 77.3 68.0 / 77.3 SEDT (Liu et al., 2017a) 68.1 / 77.5 68.5 / 78.0 RaSoR (Lee et al., 2016) 70.8 / 78.7 69.6 / 77.7 FastQAExt (Weissenborn et al., 2017) 70.8 / 78.9 70.8 / 78.9 ReasoNet (Shen et al., 2017b) 69.1 / 78.9 70.6…”
REALM: Retrieval-Augmented Language Model Pre-Training (2020, methodology): “Following previous reading comprehension work (Rajpurkar et al., 2016; Seo et al., 2016; Lee et al., 2016; Clark & Gardner, 2017), we will assume that the answer y can be found as a contiguous sequence of tokens in some document z.”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, background): “, 2019) multiple words from the input – particularly as pretraining for language generation tasks – SpanBERT pretrains span representations (Lee et al., 2016), which are widely used for question answering, coreference resolution and a variety of other tasks.”
Latent Retrieval for Weakly Supervised Open Domain Question Answering (2019, methodology): “Following recent advances in transfer learning, all scoring components are derived from BERT (Devlin et al., 2018), a bidirectional trans-former that has been pre-trained on unsupervised language-modeling data.”
Reading Wikipedia to Answer Open-Domain Questions (2017, background): “• Aligned question embedding : Following (Lee et al., 2016) and other recent works, the last part we incorporate is an aligned question embedding f align ( p i ) = j a i,j E ( q j ) , where the attention score a i,j captures the similarity between p i and each question words q j .”
Learning Dense Representations of Phrases at Scale (2020, methodology): “Following previous work on phrase or span representations (Lee et al., 2017; Seo et al., 2018), we ﬁrst apply a pre-trained language model M p to obtain contextualized word representations for each passage token: h 1 , . . . , h m ∈ R d .”
Zero-Shot Relation Extraction via Reading Comprehension (2017, background): “The task of obtaining answer spans to natural-language questions has been recently studied on the SQuAD dataset (Rajpurkar et al., 2016; Xiong et al., 2016; Lee et al., 2016; Wang et al., 2016).”
The Effect of Natural Distribution Shift on Question Answering Models (2020, neutral): “RaSoR (Single) (Lee et al., 2016) https://worksheets.”
Giving BERT a Calculator: Finding Operations and Arguments with Reading Comprehension (2019, background): “Text spans are scored as if they were another binary operation taking as arguments the start and end indices i and j of the span (Lee et al., 2017): hd = MLPspan(hi,hj) (2) and scored as ρ(d) = wᵀspanhd.”
Adversarial Examples for Evaluating Reading Comprehension Systems (2017, methodology): “We also validate our major ﬁndings on twelve other published models with publicly available test-time code: ReasoNet Single and Ensemble versions (Shen et al., 2017), Mnemonic Reader Single and Ensemble versions (Hu et al., 2017), Structural Embedding of Dependency Trees (SEDT) Single and Ensemble v”
Gated Self-Matching Networks for Reading Comprehension and Question Answering (2017, background): “Yu et al. (2016) and Lee et al. (2016) solve SQuAD by ranking continuous text spans within passage.”
ReasoNet: Learning to Stop Reading in Machine Comprehension (2016, neutral): “…[18] 40.4 51.0 - - Dynamic Chunk Reader [34] 62.5 71.0 - - Fine-Grained Gating [33] 62.5 73.3 - - Match-LSTM [26] 64.7 73.7 67.9 77.0 RaSoR [14] - - 67.4 75.5 Multi-Perspective Matching [28] 68 Iterative Co-attention Network α 67.5 76.8 - - FastQA [30] 68.4 77.1 70.8 78.9 jNet [36] 68.7…”
Stochastic Answer Networks for Machine Reading Comprehension (2017, methodology): “The average training time per epoch is comparable: our implementation running on a GTX Titan X is 22 minutes for 5-step memory net, 30 minutes for ReasoNet, and 24 minutes for SAN.”
End-to-end Neural Coreference Resolution (2017, methodology): “The above span information is concatenated to produce the ﬁnal representation g i of span i : This generalizes the recurrent span representations recently proposed for question-answering (Lee et al., 2016), which only include the boundary representations x ∗ START ( i ) and x ∗ END ( i ) .”
Multi-Perspective Context Matching for Machine Comprehension (2016, neutral): “Lee et al. (2016) Boundary Identiﬁcation.”
Efficient and Robust Question Answering from Minimal Context over Documents (2018, neutral): “We compare with RaSOR (Lee et al., 2016), ReasoNet (Shen et al.”
FusionNet: Fusing via Fully-Aware Attention with Application to Machine Comprehension (2017, positive): “EM / F1 LR Baseline (Rajpurkar et al., 2016) 40.4 / 51.0 Match-LSTM (Wang & Jiang, 2016) 64.7 / 73.7 BiDAF (Seo et al., 2017) 68.0 / 77.3 SEDT (Liu et al., 2017) 68.2 / 77.5 RaSoR (Lee et al., 2016) 70.8 / 78.7 DrQA (Chen et al., 2017a) 70.7 / 79.4 ReasoNet (Shen et al., 2017) 70.6 / 79.4 R. Mnemoni”