Siamese CBOW: Optimizing Word Embeddings for Sentence Representations

By Tom Kenter, Alexey Borisov, Maarten de Rijke. 2016

274 citations recorded by the source corpus.

We present the Siamese Continuous Bag of Words (Siamese CBOW) model, a neural network for efficient estimation of high-quality sentence embeddings. Averaging the embeddings of words in a sentence has proven to be a surprisingly successful and efficient way of obtaining sentence embeddings. However, word embeddings trained with the methods currently available are not optimized for the task of sentence representation, and, thus, likely to be suboptimal. Siamese CBOW handles this problem by training word embeddings directly for the purpose of being averaged. The underlying neural network learns word embeddings by predicting, from a sentence representation, its surrounding sentences. We show the robustness of the Siamese CBOW model by evaluating it on 20 datasets stemming from a wide variety of sources.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

An efficient framework for learning sentence representations (2018, background): “Kenter et al. (2016) learn bag-of-words (BoW) representations of sentences by considering a conceptually similar task of identifying context sentences from candidates and evaluate their representations on sentence similarity tasks.”
SemEval-2017 Task 1: Semantic Textual Similarity Multilingual and Crosslingual Focused Evaluation (2017, background): “…semantic representations (i.a., Arora et al. (2017); Conneau et al. (2017); Mu et al. (2017); Pagliardini et al. (2017); Wieting and Gimpel (2017); He and Lin (2016); Hill et al. (2016); Kenter et al. (2016); Lau and Baldwin (2016); Wieting et al. (2016a,b); He et al. (2015); Pham et al. (2015)).”
Webly Supervised Joint Embedding for Cross-Modal Image-Text Retrieval (2018, background): “Averaged word vectors has been shown to be a strong feature for text in several tasks [26, 27, 58].”
Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (2017, neutral): “C-BOW (Mikolov et al., 2013b;a) tries to predict a chosen target word given its ﬁxed-size context window, the context being deﬁned by the average of the vectors associated with the words at a distance less than the window size hyper-parameter ws .”
Learning Distributed Representations of Texts and Entities from Knowledge Base (2017, methodology): “The task has been used as a standard method to evaluate the quality of distributed representations of sentences in past work (Kiros et al., 2015; Hill et al., 2016a; Kenter et al., 2016).”
Seeping Semantics: Linking Datasets Using Word Embeddings for Data Discovery (2018, background): “Of those, averaging word embeddings (AWE) is quite popular [13], [14], [15].”
Domain-specific Knowledge Extraction from the Web of Data (2018, methodology): “This simple averaging method has proven to be a strong baseline for multiple tasks [Kenter et al. 2016].”
Deep Learning--based Text Classification (2020, background): “[123] propose a Siamese CBOW model, which forms a sentence vector representation by averaging the word embeddings of the sentence, and calculates the sentence similarity as cosine similarity between sentence vectors.”
Representing Sentences as Low-Rank Subspaces (2017, methodology): “Another simple and common approach ignores the latent structure of sentences: a prototypical approach is to represent a sentence by summing or averaging over the vectors of the words in this sentence (Wieting et al., 2015; Adi et al., 2016; Kenter et al., 2016).”
Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations (2018, methodology): “GloVe vectors; average Siamese-CBOW embeddings (Kenter et al., 2016) based on the Toronto Book Corpus; Sent2Vec (Pagliardini et al.”
Rethinking Skip-thought: A Neighborhood based Approach (2017, background): “Previously, the skip-thought model was introduced by Kiros et al. (2015), which learns to explore the semantic continuity within adjacent sentences (Harris, 1954) as supervision for learning a generic sentence encoder.”
Trimming and Improving Skip-thought Vectors (2017, background): “Previously, [12] proposed the continuous bag-of-words (CBOW) model and the skip-gram model for distributed word representation learning.”