Efficient Non-parametric Estimation of Multiple Embeddings per Word in Vector Space

By Arvind Neelakantan, Jeevan Shankar, Alexandre Passos, Andrew McCallum. 2014

474 citations recorded by the source corpus.

There is rising interest in vector-space word embeddings and their use in NLP, especially given recent methods for their fast estimation at very large scale. Nearly all this work, however, assumes a single vector per word type—ignoring polysemy and thus jeopardizing their usefulness for downstream tasks. We present an extension to the Skip-gram model that efficiently learns multiple embeddings per word type. It differs from recent related work by jointly performing word sense discrimination and embedding learning, by non-parametrically estimating the number of senses per word type, and by its efficiency and scalability. We present new state-of-the-art results in the word similarity in context task and demonstrate its scalability by training with one machine on a corpus of nearly 1 billion tokens in less than 6 hours.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Deep Contextualized Word Representations (2018, methodology): “Previously proposed methods overcome some of the shortcomings of traditional word vectors by either enriching them with subword information (e.g., Wieting et al., 2016; Bojanowski et al., 2017) or learning separate vectors for each word sense (e.g., Neelakantan et al., 2014).”
WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations (2018, background): “…around this limitation dozens of proposals have been put forward, mainly in two categories: multi-prototype embeddings (Reisinger and Mooney, 2010; Neelakantan et al., 2014; Pelevina et al., 2016), which usually leverage context clustering in order to learn distinct representations for individual…”
Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks (2019, background): “In a similar vein, we can similarly motivate this by considering the multi-sense nature of natural language (Li and Jurafsky, 2015; Neelakantan et al., 2015; Huang et al., 2012).”
Learning to Represent Words in Context with Multilingual Supervision (2015, background): “…the qualityof our representations as features in three downstream tasks: prediction of semantic supersenses (which assign nouns and verbs into a few dozen semantic classes), low res urce machine translation, and a lexical substitution task, and obtain state-of-the-art results on all of these.”
Making Sense of Word Embeddings (2016, background): “Several architectures for learning multiprototype embeddings were proposed that try to address this shortcoming (Huang et al., 2012; Tian et al., 2014; Neelakantan et al., 2014; Nieto Piña and Johansson, 2015; Bartunov et al., 2016).”
Neural information retrieval: at the end of the early years (2017, background): “2014), or something else (such as count-based embeddings)? Can embeddings from multiple sets be selected between dynamically or combined together (Neelakantan et al. 2014; Zhang et al. 2016b)? Corpora What training data/corpora should be used? Does performance vary”
Linear Algebraic Structure of Word Senses, with Applications to Polysemy (2016, result): “The results are compared to those of existing embedding based approaches reported in related work (Huang et al., 2012; Neelakantan et al., 2014; Mu et al., 2017).”
Neural Lattice Language Models (2018, background): “While many strategies for choosing the number of embeddings exist in the literature (Neelakantan et al., 2014), in this work, we choose a number of embeddings E and assign that many embeddings to each word.”
From Word to Sense Embeddings: A Survey on Vector Representations of Meaning (2018, methodology): “For instance, the model of Huang et al. (2012) took around one week to learn sense embeddings for a 6,000 subset of the 100,000 vocabulary on a corpus of one billion tokens (Neelakantan et al., 2014).”
A Simple Word Embedding Model for Lexical Substitution (2015, background): “Few recent models extended word embeddings by learning a distinct representation for each sense of a target word type, as induced by clustering the word’s contexts (Huang et al., 2012; Neelakantan et al., 2014).”
Coming to Your Senses: on Controls and Evaluation Sets in Polysemy Research (2018, methodology): “Prior art has either taken an automatic approach to detect word senses based on the statistics extracted from texts (Huang et al., 2012; Neelakantan et al., 2014; Li and Jurafsky, 2015), or relied on external lexical resources (e.g., WordNet or BabelNet) which guarantee that the detected senses are…”
Query Expansion with Locally-Trained Word Embeddings (2016, unknown): “number of senses for each word may be fixed (Neelakantan et al., 2015), or determined using class labels (Trask et al.”
Does BERT Make Any Sense? Interpretable Word Sense Disambiguation with Contextualized Embeddings (2019, background): “Neelakantan et al. (2014) first introduced a neural model to learn multiple embeddings for one word depending on different senses.”
MKD: a Multi-Task Knowledge Distillation Approach for Pretrained Language Models (2019, background): “For more ﬂexible word representations, a few advancements exist: Neelakantan et al. (2015) learn multiple embeddings per word type; context2vec (Melamud et al., 2016) uses bidirectional LSTM to encode contexts around target words; CoVe (McCann et al., 2017) trains LSTM encoders on some machine…”
Neural context embeddings for automatic discovery of word senses (2015, methodology): “The second extends an existing method for simultaneous embedding and clustering of words (Neelakantan et al., 2014).”
Reading Twice for Natural Language Understanding (2017, background): “Following studies [17, 5], however, quickly realized that a single representation can hardly account for lexical ambiguity and proposed to model words by multiple representations instead.”
Embedding Words and Senses Together via Joint Knowledge-Enhanced Training (2016, background): “…inducing word senses from monolingual corpora (Schütze, 1998; Reisinger and Mooney, 2010; Huang et al., 2012; Di Marco and Navigli, 2013; Neelakantan et al., 2014; Tian et al., 2014; Li and Jurafsky, 2015; Vu and Parker, 2016; Qiu et al., 2016), or bilingual parallel data (Guo et al.,…”