ABCNN: Attention-Based Convolutional Neural Network for Modeling Sentence Pairs

By Wenpeng Yin, Hinrich Schütze, Bing Xiang, Bowen Zhou. 2015

975 citations recorded by the source corpus.

How to model a pair of sentences is a critical issue in many NLP tasks such as answer selection (AS), paraphrase identification (PI) and textual entailment (TE). Most prior work (i) deals with one individual task by fine-tuning a specific system; (ii) models each sentence’s representation separately, rarely considering the impact of the other sentence; or (iii) relies fully on manually designed, task-specific linguistic features. This work presents a general Attention Based Convolutional Neural Network (ABCNN) for modeling a pair of sentences. We make three contributions. (i) The ABCNN can be applied to a wide variety of tasks that require modeling of sentence pairs. (ii) We propose three attention schemes that integrate mutual influence between sentences into CNNs; thus, the representation of each sentence takes into consideration its counterpart. These interdependent sentence pair representations are more powerful than isolated sentence representations. (iii) ABCNNs achieve state-of-the-art performance on AS, PI and TE tasks. We release code at: https://github.com/yinwenpeng/Answer_Selection.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

A Decomposable Attention Model for Natural Language Inference (2016, methodology): “The neural counterpart to alignment, attention (Bahdanau et al., 2015), which is a key part of our approach, was originally proposed and has been predominantly used in conjunction with LSTMs (Rockt¨aschel et al., 2016; Wang and Jiang, 2015) and to a lesser extent with CNNs (Yin et al., 2016).”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “r to A. Zhong et al. [2020] showed that a BERT-based model performs better a tf–idf similarity model in this task, but is slightly less effective than an attention-based convolutional neural network [Yin et al., 2016]. There are relatively few transform-based approaches to legal information retrieva”
Analysis Methods in Neural Language Processing: A Survey (2018, background): “For instance, in attacking malware detection systems, several studies developed targeted attacks in a black-box scenario (Yuan et al., 2017).”
Hierarchical Question-Image Co-Attention for Visual Question Answering (2016, methodology): “Also focused on modeling sentence pairs, the authors in [24] propose an attention-based bigram CNN for jointly performing attention between two CNN hierarchies.”
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks (2016, background): “In NLP tasks, multi-task learning has the potential to improve not only higher-level tasks, but also lower-level tasks.”
Joint Embedding of Words and Labels for Text Classification (2018, methodology): “…usually consist of an encoder and a decoder connected through an attention mechanism (Vaswani et al., 2017; Bahdanau et al., 2015), with successful applications to sentiment classiﬁcation (Zhou et al., 2016), sentence pair modeling (Yin et al., 2016) and sentence summarization (Rush et al., 2015).”
Attention-based LSTM for Aspect-level Sentiment Classification (2016, background): “Neural networks have achieved state-of-the-art performance in a variety of NLP tasks such as machine translation (Lample et al., 2016), paraphrase identiﬁcation (Yin et al., 2015), question answering (Golub and He, 2016) and text summarization (Rush et al., 2015).”
Efficient and Robust Question Answering from Minimal Context over Documents (2018, background): “Sentence selection The task of selecting sentences that can answer to the question has been studied across several QA datasets (Yang et al., 2015), by modeling relevance between a sentence and the question (Yin et al., 2016; Miller et al., 2016; Min et al., 2017).”
Explainable Prediction of Medical Codes from Clinical Text (2018, background): “Other recent work has combined convolution and attention (e.g., Allamanis et al., 2016; Yin et al., 2016; dos Santos et al., 2016; Yin and Schütze, 2017).”
Bilateral Multi-Perspective Matching for Natural Language Sentences (2017, methodology): “Therefore, many neural network models were proposed to match sentences from multiple level of granularity [Yin et al. , 2015; Wang and Jiang, 2016; Wang et al. , 2016d].”
Effective Parallel Corpus Mining using Bilingual Sentence Embeddings (2018, background): “Gr´egoire and Langlais (2017) use a Siamese network (Yin et al., 2015) to map source and target language sentences into the same space, then classify whether the sentences are parallel based on labelled data.”
A Context-Aware User-Item Representation Learning for Item Recommendation (2017, background): “For example, several works propose to utilize an attention layer to learn context-based representation for question and answer matching [29, 30, 31, 32].”
Adaptive Convolutional Filter Generation for Natural Language Understanding (2017, methodology): “Baselines We also compare our ACNN models with twelve strong baseline methods, categorized as follows: (i) Word Cnt, Wgt Word Cnt, LCLR (Yang, Yih, and Meek, 2015): these methods leverage lexical semantic features for prediction; (ii) Paragraph vector (PV) (Le and Mikolov, 2014): an unsupervised alg”
Question Answering and Question Generation as Dual Tasks (2017, background): “In recent years, neural network approaches (Hu et al. 2014; Yu et al. 2014; Yin et al. 2016) show promising ability in modeling the semantic relation between sentences and achieve strong performances on QA tasks.”
Multi-Matching Network for Multiple Choice Reading Comprehension (2019, background): “For example, (Yin et al. 2016) proposed to match passages against sequences that concatenate both questions and candidate answers; (Dhingra et al. 2017; Chen, Bolton, and Man-ning 2016; Lai et al. 2017; Zhu et al. 2018) proposed to ﬁrst match passages to questions and then select answers based on…”