Pairwise Word Interaction Modeling with Deep Neural Networks for Semantic Similarity Measurement

By Hua He, Jimmy J. Lin. 2016

246 citations recorded by the source corpus.

Textual similarity measurement is a challenging problem, as it requires understanding the semantics of input sentences. Most previous neural network models use coarse-grained sentence modeling, which has difficulty capturing fine-grained word-level information for semantic comparisons. As an alternative, we propose to explicitly model pairwise word interactions and present a novel similarity focus mechanism to identify important correspondences for better similarity measurement. Our ideas are implemented in a novel neural network architecture that demonstrates state-ofthe-art accuracy on three SemEval tasks and two answer selection tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, background): “…semantic representations (i.a., Arora et al. (2017); Conneau et al. (2017); Mu et al. (2017); Pagliardini et al. (2017); Wieting and Gimpel (2017); He and Lin (2016); Hill et al. (2016); Kenter et al. (2016); Lau and Baldwin (2016); Wieting et al. (2016a,b); He et al. (2015); Pham et al. (2015)).”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, methodology): “Although other distance metrics can be used as well, for example, see He and Lin [2016], Pang et al. [2016]. (27)One might argue that, with this class of models, we have simply replaced feature engineering (from learning to rank) with network engineering, since there are pretty clear analogies betwe”
Distilling Task-Specific Knowledge from BERT into Simple Neural Networks (2019, background): “For sentence-pair tasks, the siamese counterpart uses no pairwise word interactions, unlike previous state of the art (He and Lin, 2016); its runtime thus scales linearly with sentence length.”
Learning protein sequence embeddings using information from structure (2019, background): “Further elaborate directional alignments have been used for question answering and reading comprehension models [25, 26, 27] and for natural language inference [28, 29, 29].”
HCTI at SemEval-2017 Task 1: Use convolutional neural network to evaluate Semantic Textual Similarity (2017, background): “However, neural network systems (He et al., 2015a; He and Lin, 2016) can alleviate data sparseness with pre-training and distributed representations.”
Semantic Sentence Matching with Densely-connected Recurrent and Co-attentive Information (2018, methodology): “To encourage gradient to flow in the backward pass, residual connection (He et al. 2016) is introduced which bypasses the non-linear transformations with an identity mapping.”
UMD-TTIC-UW at SemEval-2016 Task 1: Attention-Based Multi-Perspective Convolutional Neural Networks for Textual Similarity Measurement (2016, background): “Competitive systems in recent years are mostly based on neural networks (He et al., 2015; Tai et al., 2015; Yin and Sch¨utze, 2015; He and Lin, 2016), which can alleviate data sparseness with pre-training and distributed representations.”
A Compare-Aggregate Model for Matching Text Sequences (2016, methodology): “He & Lin (2016) proposed a pairwise word interaction model that ﬁrst takes each pair of words from two sequences and applies a comparison unit on the two words.”
Pervasive Attention: 2D Convolutional Neural Networks for Sequence-to-Sequence Prediction (2018, background): “They also observed that such a structure implements an implicit form of attention, by producing an input encoding that depends on the output sequence produced so far.”
CoaCor: Code Annotation for Code Retrieval with Reinforcement Learning (2019, background): “, previous work on NL paraphase detection [16, 26]) are applicable and can be explored as future work.”
SimGNN: A Neural Network Approach to Fast Graph Similarity Computation (2018, background): “An analogy is that, in Natural Language Processing, the performance of sentence matching based on one embedding per sentence can be further enhanced through using fine-grained word-level information [19, 21].”
Alignment over Heterogeneous Embeddings for Question Answering (2019, methodology): “Most use stacked architectures usually coupled with attention mechanisms (He and Lin, 2016; Yin et al., 2015; Seo et al., 2016; Xiong et al., 2016b; Kumar et al., 2016; Tan et al., 2015; Wang et al., 2017a; Chen et al., 2016; Cheng et al., 2016; Golub and He, 2016).”
Automatically Extracting High-Quality Negative Examples for Answer Selection in Question Answering (2017, background): “In recent years, researchers have had substantial success in tack-ling the answer selection problem with neural networks, e.g., [6, 7, 10, 11, 17]. continuous representations that deep-learning approaches provide are effective in combating data sparsity, a per-petual challenge in natural language…”
Evolution of Semantic Similarity—A Survey (2020, methodology): “• He and Lin[30] proposed a hybrid architecture using Bi-LSTM and CNN to estimate the semantic similarity of the model.”
Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Tasks (2020, methodology): “A fine-tuned BERT [DCLT18], a LSTM+Attention model [CZL17], and a LSTM+CNN model [HL16] are used to compare the performance of training with the square loss and the cross-entropy loss.”
Improved Semantic-Aware Network Embedding with Fine-Grained Word Alignment (2018, background): “cult to accomplish in models where text sequences are first embedded into a common space and then compared in pairs (He and Lin, 2016; Parikh et al., 2016; Wang and Jiang, 2017; Wang et al., 2017b; Shen et al., 2018a).”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, result): “This can be mitigated by choosing activation functions, such as the Rectiﬁed Linear Unit (ReLU) [44], which do not exhibit regions that are arˆetically steep or have bosonically small gradients.”