A Compare-Aggregate Model for Matching Text Sequences

By Shuohang Wang, Jing Jiang. 2016

280 citations recorded by the source corpus.

Many NLP tasks including machine comprehension, answer selection and text entailment require the comparison between sequences. Matching the important units between sequences is a key to solve these problems. In this paper, we present a general "compare-aggregate" framework that performs word-level matching followed by aggregation using Convolutional Neural Networks. We particularly focus on the different comparison functions we can use to match two vectors. We use four different datasets to evaluate the model. We find that some simple comparison functions based on element-wise operations can work better than standard neural network and neural tensor network.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Dual Co-Matching Network for Multi-choice Reading Comprehension (2019, methodology): “Previous work in (Tai et al., 2015; Wang and Jiang, 2016) shows this method can build better matching representation.”
R3: Reinforced Reader-Ranker for Open-Domain Question Answering (2017, methodology): “…representations M ∈ R 2 l × P using H p and H q as follows: where W m ∈ R 2 l × 4 l are learnable parameters; · · is the column concatenation of matrices; Element-wise operations ( · · ) and ( · − · ) are also used to represent word-level matching (Wang and Jiang, 2017a; Chen et al., 2017b).”
A Co-Matching Model for Multi-choice Reading Comprehension (2018, methodology): “Inspired by some previous work (Wang and Jiang, 2016; Trischler et al., 2016), we ﬁrst use bi-directional LSTMs (Hochre-iter and Schmidhuber, 1997) to pre-process the sequences as follows: where H p ∈ R l × P , H q ∈ R l × Q and H a ∈ R l × A are the sequences of hidden states generated by the…”
Bilateral Multi-Perspective Matching for Natural Language Sentences (2017, methodology): “Therefore, many neural network models were proposed to match sentences from multiple level of granularity [Yin et al., 2015; Wang and Jiang, 2016; Wang et al., 2016d].”
Learning to Rank Question-Answer Pairs Using Hierarchical Recurrent Encoder with Latent Topic Clustering (2017, methodology): “Models [1,3-6] are from (Lowe et al., 2015; Wang and Jiang, 2016; Wang et al., 2017; Baudiš et al., 2016; Tan et al., 2015), respectively.”
Quick and (not so) Dirty: Unsupervised Selection of Justification Sentences for Multi-hop Question Answering (2019, methodology): “…methods have achieved high performance on some of these datasets (Sun et al., 2018; Trivedi et al., 2019; Tymoshenko et al., 2017; Seo et al., 2016; Wang and Jiang, 2016; De Cao et al., 2018; Back et al., 2018), we argue that more effort must be dedicated to explaining their inference process. ar…”
DiPair: Fast and Accurate Distillation for Trillion-ScaleText Matching and Pair Modeling (2020, methodology): “Early work like (Wang and Jiang, 2017) uses attention to aggregate the two sequences of word embeddings, and a CNN model is then applied to extract the final representation.”
The Cascade Transformer: an Application for Efficient Answer Sentence Selection (2020, background): “The use of compare and aggregate architectures has also been extensively evaluated (Wang and Jiang, 2016; Bian et al., 2017; Yoon et al., 2019).”
Reranking for Efficient Transformer-based Answer Selection (2020, methodology): “In particular, we use (i) simple models, e.g. , Jaccard similarity, as well as light neural models such as Compare-Aggregate [22], for reranking answer sentence candidates; and (ii) BERT models as our final AS2 step.”
Multi-range Reasoning for Machine Comprehension (2018, methodology): “More speciﬁcally, we use the ‘SubMultNN’ or the ‘’Mult’ adaptation from [30] (this is tuned) and compare aligned sequences between P and Q to form P q , the query-dependent passage representation.”
Improved Semantic-Aware Network Embedding with Fine-Grained Word Alignment (2018, methodology): “Inspired by the observation in Wang and Jiang (2017) that simple comparison/alignment functions based on element-wise operations exhibit excellent performance in matching text sequences, here we use a combination of element-wise sub-traction and multiplication as where denotes the element-wise…”
TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection (2019, methodology): “ally deﬁned in the TREC competition (Wang, Smith, and Mitamura 2007). With the advent of neural models, it has had signiﬁcant contributions from techniques such as (He and Lin 2016; Yang et al. 2018; Wang and Jiang 2016). Recently, approaches for training neural language models, e.g., ELMO (Peters e”
Semantic Text Matching for Long-Form Documents (2019, background): “The Siamese structure associated with two identical sub-networks were shown to be effective in measuring the affinity between representations of two documents modeled in the same hidden space [32, 38, 47, 54].”
Distilling Knowledge for Fast Retrieval-based Chat-bots (2020, methodology): “However, for comparing the last layer encodings we use a combination of scaled dot-product attention [18] and the SubMult function [19] for calculating the matching score.”
Quora Question Pairs (2017, methodology): “To cope with the limitations of the siamese framework, a second framework named “compare-aggregate” is proposed [4], which captures the interaction between two sentences by performing a word-level matching and aggregating the matching results into a vector for the ﬁnal classiﬁcation.”