Ruminating Reader: Reasoning with Gated Multi-hop Attention

By Yichen Gong, Samuel R. Bowman. 2017

48 citations recorded by the source corpus.

To answer the question in machine comprehension (MC) task, the models need to establish the interaction between the question and the context. To tackle the problem that the single-pass model cannot reflect on and correct its answer, we present Ruminating Reader. Ruminating Reader adds a second pass of attention and a novel information fusion component to the Bi-Directional Attention Flow model (BiDAF). We propose novel layer structures that construct a query aware context vector representation and fuse encoding representation with intermediate representation on top of BiDAF model. We show that a multi-hop attention mechanism can be applied to a bi-directional attention structure. In experiments on SQuAD, we find that the Reader outperforms the BiDAF baseline by 2.1 F1 score and 2.7 EM score. Our analysis shows that different hops of the attention have different responsibilities in selecting answers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension (2018, neutral): “To measure the speedup of our model against the RNN models, we also test the corresponding model architecture with each encoder block replaced with a stack of bidirectional Published 12 LeaderBoard 13 Single Model EM / F1 EM / F1 LR Baseline (Rajpurkar et al., 2016) 40.4 / 51.0 40.4 / 51.0 Dynamic C”
The Effect of Natural Distribution Shift on Question Answering Models (2020, methodology): “Hendrycks et al. (2020) evaluates robustness to distribution shift for question answering, among other tasks, by carefully splitting subsets of the ReCoRD dataset (Zhang et al.”
Adversarial Examples for Evaluating Reading Comprehension Systems (2017, background): “We also validate our major ﬁndings on twelve other published models with publicly available test-time code: ReasoNet Single and Ensemble versions (Shen et al., 2017), Mnemonic Reader Single and Ensemble versions (Hu et al., 2017), Structural Embedding of Dependency Trees (SEDT) Single and Ensemble v”
Natural Language Inference over Interaction Space (2017, methodology): “Many variants of attention technique such as hard-attention(Xu et al., 2015), self-attention(Parikh et al., 2016), multi-hop attention(Gong & Bowman, 2017), bidirectional attention(Seo et al., 2016) and multi-head attention(Vaswani et al., 2017) are also introduced to tackle more complicated tasks.”
Paragraph-level Neural Question Generation with Maxout Pointer and Gated Self-attention Networks (2018, methodology): “It has two steps: 1) taking encoded passage-answer representation u as input and conducting matching against itself to compute self matching representation; (Wang et al., 2017b) 2) combining the input with self matching representation using a feature fusion gate (Gong and Bowman, 2017).”
Bi-Directional Block Self-Attention for Fast and Memory-Efficient Sequence Modeling (2018, methodology): “This provides us with a sequence (Gong & Bowman, 2017) is employed to combine them, and generates the ﬁnal context-aware representations of all tokens, i.e., where σ ( · ) is an activation function, and u = [ u 1 , u 2 , . . . , u n ] ∈ R d e × n is the mBloSA output, which consists of the…”
Smarnet: Teaching Machines to Read and Comprehend Like Human (2017, unknown): “Many previous approaches (Seo et al. 2016) (Gong and Bowman 2017) (Wang et al. 2017) adopt attention mechanisms along with deep neural network tactics and pointer network to establish interactions between the question and the passage.”
Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text (2018, background): “For QA over text only, a major focus has been on the task of reading comprehension (Seo et al., 2017; Gong and Bowman, 2017; Hu et al., 2017; Shen et al., 2017; Yu et al., 2018) since the introduction of SQuAD (Rajpurkar et al.”
Mnemonic Reader for Machine Comprehension (2017, methodology): “However, most of the existing methods, such as Answer Pointer (Wang and Jiang, 2017) and Ruminating Reader (Gong and Bowman, 2017), focus on the query-aware context representation and use query-independent pointer vector to select the answer boundary.”
On Knowledge distillation from complex networks for response prediction (2019, background): “…component in most state of the art models (Hermann et al., 2015; Kadlec et al., 2016; Cao et al., 2016; Xiong et al., 2016; Seo et al., 2016; Gong and Bowman, 2017; Dhingra et al., 2017; Wang et al., 2017; Shen et al., 2017; Trischler et al., 2016; Group and Asia, 2017; Tan et al., 2017;…”
Making Neural Machine Reading Comprehension Faster (2019, methodology): “ationally faster neural architecture which performs reasonably well on SQuAD task. 2 Chapter 2 Literature Review Most of the papers that approach to solve SQuAD problem have utilized RNN based models [3, 5, 7, 9, 13, 14, 16, 17, 20, 22{24, 26, 29{31, 34{39, 41, 43] and they follow a similar chain of”