Siamese Recurrent Architectures for Learning Sentence Similarity

By Jonas W. Mueller, Aditya Thyagarajan. 2016

992 citations recorded by the source corpus.

We present a siamese adaptation of the Long Short-Term Memory (LSTM) network for labeled data comprised of pairs of variable-length sequences. Our model is applied to assess semantic similarity between sentences, where we exceed state of the art, outperforming carefully handcrafted features and recently proposed neural network systems of greater complexity. For these applications, we provide word-embedding vectors supplemented with synonymic information to the LSTMs, which use a fixed size vector to encode the underlying meaning expressed in a sentence (irrespective of the particular wording/syntax). By restricting subsequent operations to rely on a simple Manhattan metric, we compel the sentence representations learned by our model to form a highly structured space whose geometry reflects complex semantic relationships. Our results are the latest in a line of findings that showcase LSTMs as powerful language models capable of tasks requiring intricate understanding.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Ethical Challenges in Data-Driven Dialogue Systems (2017, methodology): “87 .76 ; 2.61 .94) Table 3: Semantic similarity between adversarial examples and base sentences and generated responses from base sentence response. We report (cosine distance; LSTM similarity model (Mueller and Thyagarajan 2016); CNN similarity model (He, Gimpel, and Lin 2015)) scores. Ratings are ”
Do Not Have Enough Data? Deep Learning to the Rescue! (2019, background): “Generative vs. Discriminative Generally speaking, training a generative model requires more data than training a discriminative model (Ng and Jordan 2002).”
Deep Weighted Averaging Classifiers (2018, methodology): “This idea is also related to models which use neural networks to learn a similarity function for specific applications, such as face recognition [7] or text similarity [33]; here we show how this is a more generally applicable way to train models, and we emphasize the connection to interpretability.”
Learning protein sequence embeddings using information from structure (2019, background): “Word embeddings [5, 6, 7], and more recently, sentence embeddings [8, 9, 10, 11], have been shown to be invaluable as features for problems ranging from machine translation to natural language inference.”
Learning and Refining of Privileged Information-Based RNNs for Action Recognition from Depth Sequences (2017, background): “Recurrent Neural Network RNNs are neural networks with feedback loops that produce the recurrent connection in the unfolded network [6, 33, 28].”
Argumentative Relation Classification as Plausibility Ranking (2019, methodology): “The ﬁrst neural approach for ARC (Cocarascu and Toni, 2017) proposes a neural network with a Siamese structure (Koch et al., 2015; Mueller and Thyagarajan, 2016; Cocarascu and Toni, 2017).”
Neural Snowball for Few-Shot Relation Learning (2019, neutral): “(Riloff et al., 1999; Etzioni et al., 2005; Pantel and Pennacchiotti, 2006; Rozenfeld and Feldman, 2008; Nakashole et al., 2011).”
Quora Question Pairs (2017, methodology): “We then elaborate on our feature engineering process, discuss the choice of neural networks (RNNs and LSTMs) in modeling the question pairs, and propose a new siamese LSTM structure based on the existing Manhatten LSTM model [7].”
LISA: Explaining Recurrent Neural Network Judgments via Layer-wIse Semantic Accumulation and Example to Pattern Transformation (2018, background): “, 2015), semantic textual similarity (Mueller and Thyagarajan, 2016; Gupta et al., 2018a) and dynamic topic modeling (Gupta et al.”
CAIL2019-SCM: A Dataset of Similar Case Matching in Legal Domain (2019, background): “, 1994) for metric learning achieve great success and is widely applied (Amiri et al., 2016; Liu et al., 2018; Mueller and Thyagarajan, 2016; Neculoiu et al., 2016; Wan et al., 2016; He et al., 2015).”
Submodular Optimization-based Diverse Paraphrasing and its Effectiveness in Data Augmentation (2019, background): “SiameseLSTM: Siamese adaptation of LSTM to measure quality between two sentences (Mueller and Thyagarajan, 2016)”
UMDeep at SemEval-2017 Task 1: End-to-End Shared Weight LSTM Model for Semantic Textual Similarity (2017, background): “(Mueller and Thyagarajan, 2016; Tai et al., 2015) Shared Parameters.”
A Study of Various Text Augmentation Techniques for Relation Classification in Free Text (2019, methodology): “We followed the work by Mueller and Thyagarajan (Mueller and Thyagarajan, 2016) where they randomly replaced words with their synonyms obtained from WordNet Synonym Dictionary (Miller, 1995).”
Semantic Text Matching for Long-Form Documents (2019, methodology): “Under the Siamese structure [32], each SMASH RNN has two multidepth attention-based hierarchical RNN (MASH RNN) towers.”
Multi-Level Matching and Aggregation Network for Few-Shot Relation Classification (2019, methodology): “Some sentence matching methods mainly rely on sentence encoding (Mueller and Thyagarajan, 2016; Conneau et al., 2017; Chen et al., 2018), which encode a pair sentences independently and then transmit their embeddings into a classiﬁer, such as a neural network, to decide the relationship between…”
A Survey on Text Classification: From Traditional to Deep Learning (2020, background): “[216] present a siamese adaptation of the LSTM network comprised of couples of variable-length sequences.”
Deep Metric Learning: A Survey (2019, methodology): “T give an example, Muel r an Thyagaraja used the Siamese network with LSTM arc itecture to identify the semantic similarities between the sentences [53].”