Learning word embeddings efficiently with noise-contrastive estimation

By Andriy Mnih, Koray Kavukcuoglu. 2013

641 citations recorded by the source corpus.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Using the Output Embedding to Improve Language Models (2016, background): “The output embedding is typically ignored, although (Mitra et al., 2016; Mnih and Kavukcuoglu, 2013) use both the output and input embeddings of words in order to compute word similarity.”
GloVe: Global Vectors for Word Representation (2014, background): “Mnih and Kavukcuoglu (2013) also proposed closely-related vector log-bilinear models, vLBL and ivLBL, and Levy et al. (2014) proposed explicit word embed-dings based on a PPMI metric.”
Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval (2015, methodology): “Popular methods for learning word representations include (Collobert et al., 2011; Mikolov et al., 2013c; Mnih and Kavukcuoglu, 2013; Pennington et al., 2014): all are based on unsupervised objec-4 tives of predicting words or word frequencies from raw text.”
Learning Word Vectors for 157 Languages (2018, methodology): “A vector representation h of this context is obtained by averaging the corresponding word vectors: Here, we propose to use the model with position weights introduced by Mnih and Kavukcuoglu (2013).”
Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval (2020, methodology): “These methods are also rooted in Noise Constructive Estimation [30, 42], but their technical choices are different from ANCE as visual representation learning does not have a natural query and sparse retrieval to start with.”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, methodology): “This loss is called InfoNCE [49] as it is inspired by Noise-Contrastive Estimation [24, 45] and has been shown to maximize the mutual information between c i,j and z i + k,j [49].”
Contrastive Multiview Coding (2019, methodology): “Similar to recent setups for contrastive learning [56,24,50], we train this function to correctly select a single positive sample x out of a set S = {x, y1, y2, .”
Unsupervised Feature Learning via Non-parametric Instance Discrimination (2018, methodology): “We follow [25], treating it as a constant and estimating its value via Monte Carlo approximation: where { j k } is a random subset of indices.”
Flow Contrastive Estimation of Energy-Based Models (2019, methodology): “[51, 50] applied NCE to log-bilinear models and in [70] NCE is applied to neural probabilistic language models.”
Learning deep representations by mutual information estimation and maximization (2018, methodology): “5 follows NCE as used in the language modeling community (Mnih & Kavukcuoglu, 2013), which replaces binary classiﬁcation with a related set of multiclass problems.”
On Using Very Large Target Vocabulary for Neural Machine Translation (2014, methodology): “This has been used proposed recently in (Mnih and Kavukcuoglu, 2013; Mikolov et al., 2013) based on noise-contrastive estimation (Gut-mann and Hyvarinen, 2010).”
Pointing the Unknown Words (2016, neutral): “The hierarchical softmax (Morin and Bengio, 2005), importance sampling (Bengio and Sen´ecal, 2008; Jean et al., 2014), and the noise contrastive estimation (Gutmann and Hyv¨arinen, 2012; Mnih and Kavukcuoglu, 2013) methods are in the class.”
Advances in Pre-Training Distributed Word Representations (2017, methodology): “…known modiﬁcations and data pre-processing strategies that are rarely used together: the position dependent features introduced by Mnih and Kavukcuoglu (2013), the phrase representations used in Mikolov et al. (2013b) and the use of subword information (Bojanowski et al., 2017).”
Machine Learning Methods That Economists Should Know About (2019, background): “These attempt to capture latent semantic structure in language; see [Mnih and Hinton, 2007, Mnih and Teh, 2012, Mikolov et al., 2013a,b,c, Mnih and Kavukcuoglu, 2013, Pennington et al., 2014, Levy and Goldberg, 2014, Vilnis and McCallum, 2015, Arora et al., 2016, Barkan, 2016, Bamler and Mandt,…”
Direct Output Connection for a High-Rank Language Model (2018, methodology): “Moreover, such word embedding methods as Skip-gram (Mikolov et al., 2013) and vLBL (Mnih and Kavukcuoglu, 2013) originated from neural language models designed to handle much larger vocabulary and data sizes.”
SHOPPER: A Probabilistic Model of Consumer Choice with Substitutes and Complements (2017, background): “…capturing latent semantic structure in language (Mnih and Hinton, 2007; Mnih and Teh, 2012; Mikolov et al., 2013b,a; Mikolov, Yih and Zweig, 2013; Mnih and Kavukcuoglu, 2013; Pennington, Socher and Manning, 2014; Levy and Goldberg, 2014; Vilnis and McCallum, 2015; Arora et al., 2016; Barkan,…”
Deep Learning: Methods and Applications (2014, methodology): “In order to apply NCE to train NNLMs efficiently, Mnih and Teh [255] and Mnih and Kavukcuoglu [254] first formulate the learning problem as one which takes the objective function as the distribution of the word in terms of a scoring function.”
Deep Graph Infomax (2018, methodology): “Contrastive methods are central to many popular word-embedding methods (Collobert & Weston, 2008; Mnih & Kavukcuoglu, 2013; Mikolov et al., 2013), but they are found in many unsupervised algorithms for learning representations of graph-structured input as well.”