Inner Attention based Recurrent Neural Networks for Answer Selection

By Bingning Wang, Kang Liu, Jun Zhao. 2016

195 citations recorded by the source corpus.

Attention based recurrent neural networks have shown advantages in representing natural language sentences (Hermann et al., 2015; Rockt¨aschel et al., 2015; Tan et al., 2015). Based on recurrent neural networks (RNN), external attention information was added to hidden representations to get an attentive sentence representation. Despite the improvement over non-attentive models, the attention mechanism under RNN is not well studied. In this work, we analyze the deﬁciency of traditional attention based RNN models quantitatively and qualitatively. Then we present three new RNN models that add attention information before RNN hidden representation, which shows advantage in representing sentence and achieves new state-of-art results in answer selection task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Learning Recurrent Span Representations for Extractive Question Answering (2016, background): “Machine learned extractive question answering systems, such as the one presented here, learn a predictor function f ( q , p ) → a from a training dataset of (cid:104) q , p , a (cid:105) triples.”
Learning What is Essential in Questions (2017, neutral): “For instance, Neural Networks based models (Wang et al., 2016; Tymoshenko et al., 2016; Yin et al., 2016) implicitly compute some kind of attention.”
Bilateral Multi-Perspective Matching for Natural Language Sentences (2017, methodology): “The advantage of this framework is that sharing parameters makes the model smaller and easier to train, and the sentence vectors can be used for visualization, sentence clustering and many other purposes [ Wang et al. , 2016c ] .”
An End-to-End Model for Question Answering over Knowledge Base with Cross-Attention Combining Global Knowledge (2017, methodology): “Xu et al. (2016a; 2016b) proposed integrated systems to address KB-QA problems incorporating Wikipedia free text, in which they used multichannel CNNs to extract relations.”
A Deep Look into Neural Ranking Models for Information Retrieval (2019, methodology): “For example, IARNN utilizes attention-based RNN models with GRU to get an attentive sentence representation.”
Advances and Challenges in Conversational Recommender Systems: A Survey (2021, methodology): “Then, a matching function is used to combine the two representations and output a matching probability [63, 163, 138, 45, 174].”
A Context-Aware User-Item Representation Learning for Item Recommendation (2017, background): “Attention-based neural networks have been shown to be effective in improving performance in many tasks [1, 38, 39, 43].”
A^3: Accelerating Attention Mechanisms in Neural Networks with Approximation (2020, methodology): “Attention mechanism in natural language processing is used for translation [7], [10], question answering [8], [9], [19], [40], [41], [42], language inference [13], [43], summarization [44], [45], document classification”
Cross-topic Argument Mining from Heterogeneous Sources (2018, methodology): “For obtaining a concise representation of the sentence, we apply a BiLSTM model on the weighted input embeddings, whereas Wang et al. (2016) used a single GRU.”
Attention in Natural Language Processing (2019, background): “For instance, the concepts of inner attention [68] and word attention [41] are arguably one and the same.”
Knowledge-aware Attentive Neural Network for Ranking Question Answer Pairs (2018, methodology): “For cleaned TREC QA and WikiQA, in addition to [5], we compare our method with four more strong baselines: (1) a bigram CNN model with average pooling [14]; (2) the attentive pooling LSTM network [4]; (3) a position-aware attention based RNN [2]; (4) inner attention based RNN [10].”
Document Gated Reader for Open-Domain Question Answering (2019, methodology): “Inspired by Wang et al. [47, 51], for i th word in the document, we also concatenate the aligned question word embedding which is calculated by m 1 a ij w where w is j th question Glove embedding, and a i , j is calculated by: a”
Conditional Generative Adversarial Networks for Commonsense Machine Comprehension (2017, methodology): “Traditional method mainly focuses on employing off-the-shelf NLP tools to extract features such as POS tags and then build a feature engineering system [Wang et al. , 2016a; 2016b].”
MultiCQA: Zero-Shot Transfer of Self-Supervised Text Matching Models on a Massive Scale (2020, methodology): “This includes CNN and LSTM models with attention (Tan et al., 2016; Dos Santos et al., 2016; Wang et al., 2016; Rücklé and Gurevych, 2017), compare-aggregate approaches (Wang and Jiang, 2017; Rücklé et al., 2019a), and, more recently, transformer-based models (Hashemi et al., 2020; Yang et al.,…”