Probabilistic FastText for Multi-Sense Word Embeddings

By Ben Athiwaratkun, Andrew Gordon Wilson, Anima Anandkumar. 2018

158 citations recorded by the source corpus.

We introduce Probabilistic FastText, a new model for word embeddings that can capture multiple word senses, sub-word structure, and uncertainty information. In particular, we represent each word with a Gaussian mixture density, where the mean of a mixture component is given by the sum of n-grams. This representation allows the model to share the “strength” across sub-word structures (e.g. Latin roots), producing accurate representations of rare, misspelt, or even unseen words. Moreover, each component of the mixture can capture a different word sense. Probabilistic FastText outperforms both FastText, which has no probabilistic model, and dictionary-level probabilistic embeddings, which do not incorporate subword structures, on several word-similarity benchmarks, including English RareWord and foreign language datasets. We also achieve state-of-art performance on benchmarks that measure ability to discern different meanings. Thus, our model is the first to achieve best of both the worlds: multi-sense representations while having enriched semantics on rare words.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Knowledge Infused Learning (K-IL): Towards Deep Incorporation of Knowledge in Deep Learning (2019, methodology): “Recent embedding algorithms have emerged to create such representations such as Word2Vec (Goldberg and Levy 2014), GLoVe (Pennington, Socher, and Manning 2014), FastText (Athiwaratkun, Wilson, and Anandkumar 2018) and BERT (Devlin et al. 2018).”
Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviors (2018, methodology): “…models to learn different word representations for each sense of a word, Nguyen et al. (2017) use a Gaussian Mixture Model (Reynolds 2009) and Athiwaratkun, Wilson, and Anandkumar (2018) extend FastText word representations (Bojanowski et al. 2016) with a Gaussian Mixture Model…”
Modeling Islamist Extremist Communications on Social Media using Contextual Dimensions (2019, methodology): “Embedding algorithms including Word2Vec [43], GLoVe [44] and FastText [10] have proven to be effective for creating rich representations tuned to a specific domain.”
BERTScore: Evaluating Text Generation with BERT (2019, background): “Word embeddings (Mikolov et al., 2013; Penning-ton et al., 2014; Grave et al., 2018; Nguyen et al., 2017; Athiwaratkun et al., 2018) are dense representations of tokens that are learned by optimizing an objective that follow the distributional semantics candidate, where similar words are encouraged…”
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning (2018, unknown): “…Bojanowski, Grave, Joulin, & Mikolov, 2017), representing words as probability distributions (Vilnis & McCallum, 2015; Athiwaratkun & Wilson, 2017; Athiwaratkun, Wilson, & Anandkumar, 2018), learning word embeddings in multilingual vector spaces (Conneau, Lample, Ranzato, Denoyer, & J´egou, 2018;…”