Bayesian Neural Word Embedding
By Oren Barkan. 2016
89 citations recorded by the source corpus.
Recently, several works in the domain of natural language processing presented successful methods for word embedding. Among them, the Skip-Gram with negative sampling, known also as word2vec, advanced the state-of-the-art of various linguistics tasks. In this paper, we propose a scalable Bayesian neural word embedding algorithm. The algorithm relies on a Variational Bayes solution for the Skip-Gram objective and a detailed step by step description is provided. We present experimental results that demonstrate the performance of the proposed algorithm for word analogy and similarity tasks on six different datasets and show it is competitive with the original Skip-Gram method.
Subfields: Representation & Self-Supervised Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Representation & Self-Supervised Learning
- Stephan Mandt
- Susan Athey
- Robert Bamler
- Noam Koenigstein
- Guido W. Imbens
- Marc Najork
- Michael Bendersky
- Spurthi Amba Hombaiah
Citations in the corpus
- Machine Learning Methods That Economists Should Know About (2019, methodology): “Building on the ML literature (Candès and Recht [2009], Mazumder et al. [2010]), Athey et al.”
- SHOPPER: A Probabilistic Model of Consumer Choice with Substitutes and Complements (2017, background): “…structure in language (Mnih and Hinton, 2007; Mnih and Teh, 2012; Mikolov et al., 2013b,a; Mikolov, Yih and Zweig, 2013; Mnih and Kavukcuoglu, 2013; Pennington, Socher and Manning, 2014; Levy and Goldberg, 2014; Vilnis and McCallum, 2015; Arora et al., 2016; Barkan, 2016; Bamler and Mandt, 2017).”
- Dynamic Language Models for Continuously Evolving Content (2021, methodology): “[4] develop a dynamic skip-grammodel that combines a Bayesian version of the skip-gram model [5] with a latent time series, and Rudolph et al.”
- Scalable Attentive Sentence-Pair Modeling via Distilled Sentence Embedding (2019, background): “text and target representations that appear in many neural embedding methods (Barkan 2017; Barkan and Koenigstein 2016; Mikolov et al. 2013; Mnih and Hinton 2009).”
- ITEM2VEC: Neural item embedding for collaborative filtering (2016, methodology): “Recent progress in neural embedding methods for linguistic tasks have dramatically advanced state-ofthe-art NLP capabilities [6, 7, 8, 12].”
- Dynamic Word Embeddings via Skip-Gram Filtering (2017, methodology): “Barkan (2016) give an approximate Bayesian treatment of the model with Gaussian priors on the embeddings.”