Gated-Attention Readers for Text Comprehension

By Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William W. Cohen, Ruslan Salakhutdinov. 2016

435 citations recorded by the source corpus.

In this paper we study the problem of answering cloze-style questions over documents. Our model, the Gated-Attention (GA) Reader, integrates a multi-hop architecture with a novel attention mechanism, which is based on multiplicative interactions between the query embedding and the intermediate states of a recurrent neural network document reader. This enables the reader to build query-specific representations of tokens in the document for accurate answer selection. The GA Reader obtains state-of-the-art results on three benchmarks for this task–the CNN & Daily Mail news stories and the Who Did What dataset. The effectiveness of multiplicative interaction is demonstrated by an ablation study, and by comparing to alternative compositional operators for implementing the gated-attention.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multi-Granularity Hierarchical Attention Fusion Networks for Reading Comprehension and Question Answering (2018, methodology): “end neural network methods on the cloze-style datasets are based on this model (Hermann et al., 2015; Kadlec et al., 2016; Chen et al., 2016; Dhingra et al., 2016).”
U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, background): “…Currently, end-to-end neural network models have achieved great successes for machine reading comprehension (Seo et al. 2016; Kumar et al. 2015; Sukhbaatar et al. 2015; Cui et al. 2016; Xiong, Zhong, and Socher 2016; Dhingra et al. 2016; Shen et al. 2016; Hu et al. 2017; Wang, Yan, and Wu 2018).”
Bidirectional Attention Flow for Machine Comprehension (2016, methodology): “The next best approaches for the single model and ensemble runs, GAReader (Dhingra et al., 2016) and ReasoNet (Shen et al.”
NumNet: Machine Reading Comprehension with Numerical Reasoning (2019, neutral): “With the blooming of available large-scale MRC datasets, a great number of neural network-based MRC models have been proposed to answer questions for a given document including Attentive Reader (Kadlec et al., 2016), BiDAF (Seo et al., 2017), Interactive AoA Reader (Cui et al., 2017), Gated Attentio”
RACE: Large-scale ReAding Comprehension Dataset From Examinations (2017, background): “We refer readers to (Dhingra et al., 2016) for more details.”
Option Comparison Network for Multiple-choice Reading Comprehension (2019, methodology): “The Stanford AR (Chen et al., 2016) and GA Reader (Dhingra et al., 2017) variants used in (Lai et al., 2017) encode question and article independent of options, ignoring their correlations.”
Denoising Distantly Supervised Open-Domain Question Answering (2018, methodology): “Recently, with the development of machine reading comprehension technique (Chen et al., 2016; Dhingra et al., 2017a; Cui et al., 2017; Shen et al., 2017; Wang et al., 2017), researchers attempt to answer open-domain questions via performing reading comprehension on plain texts.”
Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, methodology): “We employ a gated-attention (GA) reader (Dhingra et al., 2016) as our base model in this work, but our framework does not make any assumptions about the base models being used.”
Gated Self-Matching Networks for Reading Comprehension and Question Answering (2017, background): “Dhingra et al. (2016) propose iteratively selecting important parts of the passage by a multi-plying gating function with the question representation.”
Cosmos QA: Machine Reading Comprehension with Contextual Commonsense Reasoning (2019, methodology): “Gated-Attention Reader (Dhingra et al., 2017) performs multi-hop atte tion between the question and a recurr nt neural network based paragraph encoding states.”
ReasoNet: Learning to Stop Reading in Machine Comprehension (2016, background): “Several recent studies try to simulate this revisit by combining the information in the query with the new information digested from previous iterations [6, 8, 13, 21, 29].”
Stochastic Answer Networks for Machine Reading Comprehension (2017, methodology): “The ﬁrst pioneering models employed a predetermined ﬁxed number of reasoning steps (Hill et al., 2016; Dhingra et al., 2016; Sordoni et al., 2016; Kumar et al., 2015).”
Attention-over-Attention Neural Networks for Reading Comprehension (2016, methodology): “Unlike the CAS Reader proposed by Cui et al. (2016), we do not assume any heuristics to our model, such as using merge functions: sum , avg etc.”
Crowdsourcing Multiple Choice Science Questions (2017, methodology): “We train the reading comprehension models on the training set with hyperparameters recommended by prior work ((On-ishi et al., 2016) for the AS Reader and (Dhingra et al., 2016) for the GA Reader), with early stopping on the validation data 6 .”
Quasar: Datasets for Question Answering by Search and Reading (2017, background): “GA (Single-Token): The GA Reader (Dhingra et al., 2017) is a multi-layer neural network which extracts a single token from the passage to answer a given query.”
Who did What: A Large-Scale Person-Centered Cloze Dataset (2016, background): “• Gated-Attention (GA) Reader: Attention Sum Reader with gated layers (Dhingra et al., 2016).”
How Much Reading Does Reading Comprehension Require? A Critical Investigation of Popular Benchmarks (2018, neutral): “Gated Attention Reader Introduced by Dhingra et al. (2017), the Gated Attention Reader (GAR)2 performs multiple hops over a passage, like MemNets.”