Iterative Alternating Neural Attention for Machine Reading

By Alessandro Sordoni, Philip Bachman, Yoshua Bengio. 2016

122 citations recorded by the source corpus.

We propose a novel neural attention architecture to tackle machine comprehension tasks, such as answering Cloze-style queries with respect to a document. Unlike previous models, we do not collapse the query into a single vector, instead we deploy an iterative alternating attention mechanism that allows a fine-grained exploration of both the query and the document. Our model outperforms state-of-the-art baselines in standard machine comprehension benchmarks such as CNN news articles and the Children's Book Test (CBT) dataset.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Bidirectional Attention Flow for Machine Comprehension (2016, methodology): “The third group (considered as variants of Memory Network (Weston et al., 2015)) repeats computing an attention vector between the query and the context through multiple layers, typically referred to as multi-hop (Sordoni et al., 2016; Dhingra et al., 2016).”
NewsQA: A Machine Comprehension Dataset (2016, background): “However, Chen et al. (2016) demonstrated that the task requires only limited reasoning and, in fact, performance of the strongest models (Kadlec et al., 2016; Trischler et al., 2016b; Sordoni et al., 2016) nearly matches that of humans.”
Semi-Supervised QA with Generative Domain-Adaptive Nets (2017, background): “Various neural models based on attention mechanisms (Wang and Jiang, 2016; Seo et al., 2016; Xiong et al., 2016; Wang et al., 2016; Dhingra et al., 2016; Kadlec et al., 2016; Trischler et al., 2016b; Sordoni et al., 2016; Cui et al., 2016; Chen et al., 2016) have been proposed to tackle the tasks of”
Gated Self-Matching Networks for Reading Comprehension and Question Answering (2017, background): “…with cloze-style datasets, several powerful deep learning models (Hermann et al., 2015; Hill et al., 2016; Chen et al., 2016; Kadlec et al., 2016; Sordoni et al., 2016; Cui et al., 2016; Trischler et al., 2016; Dhingra et al., 2016; Shen et al., 2016) have been introduced to solve this problem.”
Dynamic Coattention Networks For Question Answering (2016, methodology): “Sordoni et al. (2016) introduced an iterative neural attention model and applied it to cloze-style machine comprehension tasks.”
Constructing Datasets for Multi-hop Reading Comprehension Across Documents (2017, background): “, 2017; Liu and Perez, 2017), which is often iterated over several times (Sordoni et al., 2016; Neumann et al., 2016; Seo et al., 2017b; Hu et al., 2017) and may contain trainable stopping mechanisms (Graves, 2016; Shen et al.”
ReasoNet: Learning to Stop Reading in Machine Comprehension (2016, background): “Iterative Attention Reader, EpiReader and GA Reader are the three multi-turn reasoning models with xed reasoning steps.”
Gated-Attention Readers for Text Comprehension (2016, methodology): “Iterative reasoning has been found effective in several more recent models, including the Iterative Attentive Reader (Sordoni et al., 2016) and ReasoNet (Shen et al.”
Stochastic Answer Networks for Machine Reading Comprehension (2017, methodology): “The ﬁrst pioneering models employed a predetermined ﬁxed number of reasoning steps (Hill et al., 2016; Dhingra et al., 2016; Sordoni et al., 2016; Kumar et al., 2015).”
Attention-over-Attention Neural Networks for Reading Comprehension (2016, methodology): “We also added pretty new works to compare with (Dhingra et al., 2016; Sordoni et al., 2016; Trischler et al., 2016), which came to our attention when we are writing this paper.”
End-to-End Answer Chunk Extraction and Ranking for Reading Comprehension (2016, methodology): “An iterative alternating attention mechanism and gating strategies were proposed in (Sordoni, Bachman, and Bengio 2016) to optimize the attention through several hops.”
Embracing data abundance: BookTest Dataset for Reading Comprehension (2016, methodology): “…on the original CBT data and they can be compared with recent text-comprehension models utilizing this dataset (Hill et al., 2015; Kadlec et al., 2016; Chen et al., 2016; Sordoni et al., 2016; Dhingra et al., 2016; Trischler et al., 2016b; Weis-senborn, 2016; Cui et al., 2016b; Cui et al., 2016a).”
Multi-step Reasoning via Recurrent Dual Attention for Visual Dialog (2019, background): “The idea of multi-step reasoning has been explored in many tasks, including image classiﬁ-cation [34], text classiﬁcation [54], image generation [17], language-based image editing [6], Visual Question Answering (VQA) [53, 36], and Machine Reading Comprehension (MRC) [8, 12, 20, 44, 41, 31].”
Words or Characters? Fine-grained Gating for Reading Comprehension (2016, background): “A variety of models have been proposed to extract answers from given text (Hill et al., 2016; Kadlec et al., 2016; Trischler et al., 2016; Chen et al., 2016; Sordoni et al., 2016; Cui et al., 2016).”
Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge (2018, methodology): “In contrast to many previous complex models (Weston et al., 2015; Dhingra et al., 2017; Cui et al., 2017; Munkhdalai and Yu, 2016; Sordoni et al., 2016) that perform multi-turn reading of a story and a question before inferring the correct answer, we aim to tackle the cloze-style RC task in a way…”
Ruminating Reader: Reasoning with Gated Multi-hop Attention (2017, methodology): “The Iterative Alternative (IA) reader (Sordoni et al., 2016) produces query glimpse and document glimpse in each iterations and uses both glimpses to update recurrent state in each iteration.”