A fast and simple algorithm for training neural probabilistic language models

By Andriy Mnih, Yee Whye Teh. 2012

589 citations recorded by the source corpus.

In spite of their superior performance, neural probabilistic language models (NPLMs) remain far less widely used than n-gram models due to their notoriously long training times, which are measured in weeks even for moderately-sized datasets. Training NPLMs is computationally expensive because they are explicitly normalized, which leads to having to consider all words in the vocabulary when computing the log-likelihood gradients. We propose a fast and simple algorithm for training NPLMs based on noise-contrastive estimation, a newly introduced procedure for estimating unnormalized continuous distributions. We investigate the behaviour of the algorithm on the Penn Treebank corpus and show that it reduces the training times by more than an order of magnitude without affecting the quality of the resulting models. The algorithm is also more efficient and much more stable than importance sampling because it requires far fewer noise samples to perform well. We demonstrate the scalability of the proposed approach by training several neural language models on a 47M-word corpus with a 80K-word vocabulary, obtaining state-of-the-art results on the Microsoft Research Sentence Completion Challenge dataset.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Generating Sequences With Recurrent Neural Networks (2013, background): “In the case of softmax models, a further difficulty is the high computational cost of evaluating all the exponentials during training (although several methods have been to devised make training large softmax layers more efficient, including tree-based models [25, 23], low rank approximations [27] a”
Efficient Estimation of Word Representations in Vector Space (2013, methodology): “Architecture Accuracy [%] 4-gram [29] 39 Average LSA similarity [29] 49 Log-bilinear model [22] 54.”
Representation Learning with Contrastive Predictive Coding (2018, methodology): “Although we cannot evaluate p ( x ) or p ( x | c ) directly, we can use samples from these distributions, allowing us to use techniques such as Noise-Contrastive Estimation [12, 14, 15] and Importance Sampling [16] that are based on comparing the target value with randomly sampled negative values.”
Flow Contrastive Estimation of Energy-Based Models (2019, methodology): “[51, 50] applied NCE to log-bilinear models and in [70] NCE is applied to neural probabilistic language models.”
From captions to visual concepts and back (2014, methodology): “The noise contrastive estimation (NCE) technique is used to accelerate the training by avoiding the calculation of the exact denominator in (3) [30].”
Efficient softmax approximation for GPUs (2016, methodology): “While the original formulation still requires to compute the softmax normalization, Mnih and Teh [37] shows that good performance can be achieved even without it.”
Transfer Learning for Low-Resource Neural Machine Translation (2016, methodology): “It has a target vocabulary of 100K and is trained using noise-contrastive estimation (Mnih and Teh, 2012; Vaswani et al., 2013; Baltescu and Blunsom, 2015; Williams et al., 2015).”
Machine Learning Methods That Economists Should Know About (2019, background): “These attempt to capture latent semantic structure in language; see [Mnih and Hinton, 2007, Mnih and Teh, 2012, Mikolov et al., 2013a,b,c, Mnih and Kavukcuoglu, 2013, Pennington et al., 2014, Levy and Goldberg, 2014, Vilnis and McCallum, 2015, Arora et al., 2016, Barkan, 2016, Bamler and Mandt,…”
Compositional Fairness Constraints for Graph Embeddings (2019, methodology): “…scoring functions using contrastive learning methods that make use of a corruption distribution such as noise contrastive estimation (Dyer, 2014; Mnih & Teh, 2012) and similar variants (Bose et al., 2018), where the loss over a batch of edges E batch ⊆ E train is given by: e ∈E edge L edge ( s…”
SHOPPER: A Probabilistic Model of Consumer Choice with Substitutes and Complements (2017, background): “We note that neural language models have spawned myriad developments in other socalled word embedding methods for capturing latent semantic structure in language (Mnih and Hinton, 2007; Mnih and Teh, 2012; Mikolov et al., 2013b,a; Mikolov, Yih and Zweig, 2013; Mnih and Kavukcuoglu, 2013; Pennington,”
Deep Learning: Methods and Applications (2014, methodology): “In order to apply NCE to train NNLMs efficiently, Mnih and Teh [255] and Mnih and Kavukcuoglu [254] first formulate the learning problem as one which takes the objective function as the distribution of the word in terms of a scoring function.”
On Variational Bounds of Mutual Information (2019, neutral): “This requirement of self-normalization is a common choice when learning log-linear models and empirically has been shown not to negatively impact performance (Mnih & Teh, 2012).”
Structured Generative Models of Natural Source Code (2014, background): “Speciﬁcally, we let R ch ( α ) and b ch ( α ) be deﬁned as follows: For example, if a variable in scope has feature vector (identifier, type, how recently declared, how recently assigned) = (‘‘i’’, int, 3, 0) , then its corresponding R ch would be a context matrix-modulated sum of representations…”
Decoding with Large-Scale Neural Language Models Improves Translation (2013, methodology): “Second, we train using noise-contrastive estimation or NCE (Gutmann and Hyv¨arinen, 2010; Mnih and Teh, 2012), which does not require repeated summa-tions over the whole vocabulary.”
Convolutional Neural Network for Paraphrase Identification (2015, methodology): “In NCE, 20 noise words are sampled for each true example.”
BilBOWA: Fast Bilingual Distributed Representations without Word Alignments (2014, methodology): “Although faster methods exist for training NLMs [3, 6, 16], evaluating the cross-lingual regularization term (which is based on noisy word-alignments and in the worst case considers the O ( V 2 ) possible word translation-pairs) is still very expensive.”