Improving Distributional Similarity with Lessons Learned from Word Embeddings

By Omer Levy, Yoav Goldberg, Ido Dagan. 2015

1,402 citations recorded by the source corpus.

Recent trends suggest that neural-network-inspired word embedding models outperform traditional count-based distributional models on word similarity and analogy detection tasks. We reveal that much of the performance gains of word embeddings are due to certain system design choices and hyperparameter optimizations, rather than the embedding algorithms themselves. Furthermore, we show that these modifications can be transferred to traditional distributional models, yielding similar gains. In contrast to prior reports, we observe mostly local or insignificant performance differences between the methods, with no global advantage to any single approach over the others.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Distributed Representations of Sentences from Unlabelled Data (2016, background): “Examples include machine translation (Sutskever et al., 2014), image captioning (Mao et al., 2015) and dialogue systems (Serban et al., 2015).”
End-to-End Neural Ad-hoc Ranking with Kernel Pooling (2017, background): “Word embeddings trained from surrounding contexts, for example, word2vec [17], are considered to be the factorization of word pairs’ PMI matrix [14].”
Advances in Pre-Training Distributed Word Representations (2017, methodology): “Although a careful analysis performed in Levy et al. (2015) shows that the original word2vec is faster to train, produces more accurate models and takes signiﬁcantly less memory than the GloVe algorithm, the availability of large pre-trained GloVe models proved to be a useful resource for many…”
Understanding the Origins of Bias in Word Embeddings (2018, background): “naive, straightforward approach? The positive point-wise mutual information (PPMI) matrix is a count-based distributed representation commonly used in natural language processing (Levy et al., 2015).”
A Comprehensive Survey of Graph Embedding: Problems, Techniques, and Applications (2017, background): “graph embedding (random walk involved) in certain tasks as the latter relies on separate local context windows [147], [148].”
A Primer on Neural Network Models for Natural Language Processing (2015, background): “However, they are deeply connected to another family of algorithms which evolved in the NLP and IR communities, and that are based on matrix factorization (for a discussion see Levy & Goldberg, 2014b; Levy et al., 2015).”
Unsupervised Learning of Sentence Embeddings Using Compositional n-Gram Features (2017, background): “Using dynamic context windows is equivalent to weighing by the distance from the focus word w divided by the window size (Levy et al., 2015).”
Deep Neural Networks for Learning Graph Representations (2016, methodology): “Interestingly, as shown by the authors, the representations learned from the SVD method cannot completely outperform the representations from the PPMI matrix itself (Levy, Goldberg, and Dagan 2015).”
A Survey of Cross-lingual Word Embedding Models (2017, methodology): “(2015) ΩSVD X Inverted indexing Levy et al. (2017) LPMI ΩSVD X Levy et al.”
Neural information retrieval: at the end of the early years (2017, methodology): “Levy et al show that when these choices are applied to traditional DSMs, no consistent advantage of SGNS and GloVe is observed.”
Generalizing Translation Models in the Probabilistic Relevance Framework (2016, methodology): “Whilethis is not the newest method in this category (e.g. Pen-nington et al. [26] introduced GloVe and reported superior results), independent benchmarking provided by Levy et al. [19] shows that there is no fundamental performance difference between the recent word embedding models.”
Analogical Reasoning on Chinese Morphological and Semantic Relations (2018, result): “This result is consistent with performance of English dense and sparse vectors on MSR (morphology-only), SemEval (semantic-only), and Google (mixed) analogy datasets (Levy and Goldberg, 2014b; Levy et al., 2015).”
Embedding Words and Senses Together via Joint Knowledge-Enhanced Training (2016, background): “Word has been one of the most popular benchmarks for in-vitro evaluation of vector space models (Penning-ton et al., 2014; Levy et al., 2015).”
Attributed Social Network Embedding (2017, methodology): “To comprehensively represent a node for downstream applications, practitioners can add/concatenate the two vectors which has empirically shown to have better performance in distributed word representations [43], [44].”