Text Understanding with the Attention Sum Reader Network

By Rudolf Kadlec, Martin Schmid, Ondrej Bajgar, Jan Kleindienst. 2016

318 citations recorded by the source corpus.

Several large cloze-style context-question-answer datasets have been introduced recently: the CNN and Daily Mail news data and the Children's Book Test. Thanks to the size of these datasets, the associated text comprehension task is well suited for deep-learning techniques that currently seem to outperform all alternative approaches. We present a new, simple model that uses attention to directly pick the answer from the context as opposed to computing the answer using a blended representation of words in the document as is usual in similar models. This makes the model particularly suitable for question-answering problems where the answer is a single word from the document. Ensemble of our models sets new state of the art on all evaluated datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, methodology): “Most of the end-to-end neural network methods on the cloze-style datasets are based on this model (Hermann et al., 2015; Kadlec et al., 2016; Chen et al., 2016; Dhin-gra et al., 2016).”
Simple and Effective Multi-Paragraph Reading Comprehension (2017, methodology): “To handle this difﬁculty, we use a summed objective function similar to the one from Kadlec et al. (2016), that optimizes the sum of the probabilities of all answer spans.”
Bidirectional Attention Flow for Machine Comprehension (2016, methodology): “The second group computes the attention weights once, which are then fed into an output layer for ﬁnal prediction (e.g., Kadlec et al. (2016)).”
NumNet: Machine Reading Comprehension with Numerical Reasoning (2019, neutral): “…datasets, a great number of neural network-based MRC models have been proposed to answer questions for a given document including Attentive Reader (Kadlec et al., 2016), BiDAF (Seo et al., 2017), Interactive AoA Reader (Cui et al., 2017), Gated Attention Reader (Dhingra et al., 2017), R-Net (Wang…”
ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension (2018, background): “ASReader 5 (Kadlec et al., 2016) is a strong base-line model for cloze-style datasets such as (Her-3 mann et al., 2015; Hill et al., 2015).”
RACE: Large-scale ReAding Comprehension Dataset From Examinations (2017, background): “…of effort has been devoted to the machine comprehension task where people aim to construct a system with the ability to ∗* indicates equal contribution answer questions related to a document that it has to comprehend (Chen et al., 2016; Kadlec et al., 2016; Dhingra et al., 2016; Yang et al., 2017).”
NewsQA: A Machine Comprehension Dataset (2016, background): “As such, determining the correct answer relies mostly on recognizing textual entailment between the article and the question.”
Pointer Sentinel Mixture Models (2016, methodology): “In the above example, only January or March would be available as options, as February does not appear in the input.”
SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine (2017, methodology): “Attention Sum Reader Attention sum reader (ASR, Kadlec et al., 2016) is a variant of a pointer network (Vinyals et al., 2015) that was speciﬁ-cally constructed to solve a cloze-style question-answering task.”
Coarse-to-Fine Question Answering for Long Documents (2016, background): “Current state-of-the-art approaches for QA over documents are based on recurrent neural networks (RNNs) that encode the document and the ques-Query tion to determine the answer (Hermann et al., 2015; Chen et al., 2016; Kumar et al., 2016; Kadlec et al., 2016; Xiong et al., 2016).”
MS MARCO: A Human Generated MAchine Reading COmprehension Dataset (2016, background): “• Attention Sum Reader (AS Reader): AS Reader [21] is a simple model that uses attention to directly pick the answer from the context.”
A Discrete Hard EM Approach for Weakly Supervised Question Answering (2019, methodology): “Some models are trained with maximum marginal likelihood (MML) (Kadlec et al., 2016; Swayamdipta et al., 2018; Clark and Gardner, 2018; Lee et al., 2019), but it is unclear if it gives a meaningful improvement over the heuristics.”
The NarrativeQA Reading Comprehension Challenge (2017, methodology): “As with the AS Reader, we observed no signiﬁcant differences for varying number of chunks.”
Improving Neural Language Models with a Continuous Cache (2016, background): “…Variational LSTM (Merity et al., 2016) 100.9 - Pointer Sentinel LSTM (Merity et al., 2016) 80 This approach has been successfully applied to question answering, when the answer is contained in a given paragraph (Chen et al., 2016; Hermann et al., 2015; Kadlec et al., 2016; Sukhbaatar et al., 2015).”
Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, background): “Various neural models based on attention mechanisms (Wang and Jiang, 2016; Seo et al., 2016; Xiong et al., 2016; Wang et al., 2016; Dhingra et al., 2016; Kadlec et al., 2016; Trischler et al., 2016b; Sordoni et al., 2016; Cui et al., 2016; Chen et al., 2016) have been proposed to tackle the tasks of”
A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task (2016, neutral): “Concurrently with our paper, Kadlec et al. (2016) and Kobayashi et al. (2016) also experiment on these two datasets and report competitive results.”
Gated Self-Matching Networks for Reading Comprehension and Question Answering (2017, methodology): “…Comprehension Along with cloze-style datasets, several powerful deep learning models (Hermann et al., 2015; Hill et al., 2016; Chen et al., 2016; Kadlec et al., 2016; Sordoni et al., 2016; Cui et al., 2016; Trischler et al., 2016; Dhingra et al., 2016; Shen et al., 2016) have been introduced to…”
Dynamic Coattention Networks For Question Answering (2016, background): “Richardson et al. (2013) proposed two baselines, one that uses simple lexical features such as a sliding window to match bags of words, and another that uses word-distances between words in the question and in the document.”
Constructing Datasets for Multi-hop Reading Comprehension Across Documents (2017, background): “the correct answer. Processing (NLP). Contemporary end-to-end Reading Comprehension (RC) methods can learn to extract the correct answer span within a given text and approach human-level performance (Kadlec et al., 2016; Seo et al., 2017). However, for existing tasks, relevant information is often c”