Attentive Pooling Networks

By C. D. Santos, Ming Tan, Bing Xiang, Bowen Zhou. 2016

364 citations recorded by the source corpus.

In this work, we propose Attentive Pooling (AP), a two-way attention mechanism for discriminative model training. In the context of pair-wise ranking or classification with neural networks, AP enables the pooling layer to be aware of the current input pair, in a way that information from the two input items can directly influence the computation of each other's representations. Along with such representations of the paired inputs, AP jointly learns a similarity measure over projected segments (e.g. trigrams) of the pair, and subsequently, derives the corresponding attention vector for each input to guide the pooling. Our two-way attention mechanism is a general framework independent of the underlying representation learning, and it has been applied to both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) in our studies. The empirical results, from three very different benchmark tasks of question answering/answer selection, demonstrate that our proposed models outperform a variety of strong baselines and achieve state-of-the-art performance in all the benchmarks.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Key-Value Memory Networks for Directly Reading Documents (2016, methodology): “, 2015), the best published results on TRECQA and WIKIQA have been obtained by either convolutional neural networks (Santos et al., 2016; Yin and Schütze, 2015; Wang et al., 2016) or recurrent neural networks (Miao et al.”
Recurrent Neural Networks for Classifying Relations in Clinical Notes (2017, neutral): “This suggests that architecture with more flexible pooling models of signals such as attentive pooling networks [61] and neural attention models [27] may lead to more accurate relation classification, which will be our future work.”
Dataset and Neural Recurrent Sequence Labeling Model for Open-Domain Factoid Question Answering (2016, background): “…2014), convolution neural networks (Yih et al., 2014; Dong et al., 2015; Yin et al., 2015), attention models (Hermann et al., 2015; Yin et al., 2015; Santos et al., 2016), and memories (Graves et al., 2014; Weston et al., 2015; Kumar et al., 2016; Bordes et al., 2015; Sukhbaatar et al., 2015), etc.”
End-to-End Answer Chunk Extraction and Ranking for Reading Comprehension (2016, methodology): “We also set the maximum passage length to be 300 tokens, and pruned all the tokens after the 300-th token in the training set to save memory and speed up the training process.”
Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start (2020, background): “…al., 2016) and its followers (Wang and Jiang, 2016; Wang et al., 2017), as well as the attentive convolutional networks such as attentive pooling (dos Santos et al., 2016) and attentive convolution (Yin and Sch¨utze, 2018), and self-attentive large-scale language models like BERT (Devlin et al.,…”
Explainable Prediction of Medical Codes from Clinical Text (2018, background): “Other recent work has combined convolution and attention (e.g., Al-lamanis et al., 2016; Yin et al., 2016; dos Santos et al., 2016; Yin and Schütze, 2017).”
Embracing data abundance: BookTest Dataset for Reading Comprehension (2016, background): “…for every word in the query (Cui et al., 2016b) or use two-way context-query attention mechanism for every word in the context and the query (Cui et al., 2016a) that is similar in its spirit to models recently proposed in different domains, e.g. (dos Santos et al., 2016) in information retrieval.”
Question Generation for Question Answering (2017, methodology): “7 Question Generation for QA A typical answer sentence selection method, such as (Yin et al., 2016; Santos et al., 2016; Miller et al., 2016; Tymoshenko et al., 2016), computes the relevance score between input question Q and each answer candidate A , and selects the one with the highest relevance…”
Learning Context-Sensitive Convolutional Filters for Text Processing (2017, methodology): “…we compare it with several CNN-based sequence matching baselines, including Vanilla CNN (Jurczyk et al., 2016; Santos et al., 2017), attentive pooling networks (dos Santos et al., 2016), and ABCNN (Yin et al., 2016) (where an attention mechanism is employed over the two sentence representations).”
A Context-Aware User-Item Representation Learning for Item Recommendation (2017, methodology): “ Equation 6 for the j-th word and the k-th word in the user and item review documents respectively, n and m are the lengths of the user and item review documents respectively. Inspired by the work in [32, 33], we utilize an attentive matrix, T ∈ R f× , to derive the importance of each contextual fea”
Deep Learning--based Text Classification (2020, methodology): “Santos et al. [83] propose a two-way attention mechanism, known as Attentive Pooling (AP), for pair-wise ranking.”
Knowledge-aware Attentive Neural Network for Ranking Question Answer Pairs (2018, methodology): “Instead of learning the representations of the question and the answer separately, some recent studies exploit attention mechanisms to learn the interaction information between questions and answers, which can better focus on relevant parts of the input [2, 4, 7].”
Improving Multilingual Sentence Embedding using Bi-directional Dual Encoder with Additive Margin Softmax (2019, methodology): “We make use of 4 pooling layers consisting of max pooling, mean pooling, ﬁrst token pooling, and attention pooling [ dos Santos et al. , 2016 ] .”
Interpretable Drug Target Prediction Using Deep Neural Representation (2018, background): “Extending the one-way attention for pairwise inference, attentive pooling network [dos Santos et al. , 2016] provides a two-way attention mechanism that enables the input pairs to be aware of each other.”
Beyond [CLS] through Ranking by Generation (2020, neutral): “Considerable body of work exists on the use of NNs for this task (Feng et al., 2015; Severyn and Moschitti, 2015; Tan et al., 2016; dos Santos et al., 2016; Rao et al., 2016; Wang et al., 2017), where the most recent ones use BERT-based models that perform discrimination based on the special [ CLS…”