Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model

By Yoshua Bengio, Jean-Sébastien Senecal. 2008

262 citations recorded by the source corpus.

Previous work on statistical language modeling has shown that it is possible to train a feedforward neural network to approximate probabilities over sequences of words, resulting in significant error reduction when compared to standard baseline models based on n-grams. However, training the neural network model with the maximum-likelihood criterion requires computations proportional to the number of words in the vocabulary. In this paper, we introduce adaptive importance sampling as a way to accelerate training of the model. The idea is to use an adaptive n-gram model to track the conditional distributions produced by the neural network. We show that a very significant speedup can be obtained on standard problems.

Subfields: Artificial Intelligence, Natural Language Processing, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Optimization & Training

Citations in the corpus

Exploring the Limits of Language Modeling (2016, background): “Several approaches have been proposed to cope with the scaling issue: importance sampling (Bengio et al., 2003; Bengio & Sen´ecal, 2008), Noise Contrastive Estimation (NCE) (Gut-mann & Hyv¨arinen, 2010; Mnih & Kavukcuoglu, 2013), self normalizing partition functions (Vincent et al., 2015) or…”
Representation Learning with Contrastive Predictive Coding (2018, methodology): “though we cannot evaluate p(x) or p(xjc) directly, we can use samples from these distributions, allowing us to use techniques such as Noise-Contrastive Estimation [12, 14, 15] and Importance Sampling [16] that are based on comparing the target value with randomly sampled negative values. In the prop”
Efficient softmax approximation for GPUs (2016, methodology): “Our method is compared to: (1) the full softmax, (2) the hierarchical softmax (HSM) with frequency binning [31], (3) importance sampling [5, 4] and (4) the differentiated softmax [8].”
A fast and simple algorithm for training neural probabilistic language models (2012, methodology): “Unfortunately, the variance in the importance sampling estimates can make learning unstable, unless it is carefully controlled (Bengio & Sen´ecal, 2008).”
Montreal Neural Machine Translation Systems for WMT’15 (2015, positive): “To address this problem, we use the approach presented in (Jean et al., 2015), which is based on importance sampling (Bengio and S´en´ecal, 2008).”
BilBOWA: Fast Bilingual Distributed Representations without Word Alignments (2014, methodology): “Although faster methods exist for training NLMs [3, 5, 15], evaluating the cross-lingual regularization term (which is based on noisy word-alignments and in the worst case considers the O(V ) possible word translation-pairs) is still very expensive.”
Strategies for Training Large Vocabulary Neural Language Models (2015, background): “Target sampling builds upon the importance sampling work of Bengio and Sen´ecal (2008).”
BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies (2015, background): “In the past, a uniform distribution or the unigram distribution have been advocated as promising candidates for sampling distributions (Bengio & Sen ´ ecal, 2003; Jean et al., 2015; Bengio & Sen ´ ecal, 2008; Mnih & Teh, 2012).”
Polyglot: Distributed Word Representations for Multilingual NLP (2013, methodology): “to speed up the training procedure, either by changing the model architecture to exploit an algorithmic speedup (Mnih and Hinton, 2009; Morin and Bengio, 2005) or by estimating the error by sampling (Bengio and Senecal, 2008). (Collobert and Weston, 2008) shows that word embeddings can almost substi”
Exponential Family Embeddings (2016, methodology): “There are many extensions and variants of word embeddings (Bengio et al., 2003; 2006; Mnih & Hinton, 2007; Mikolov et al., 2013a;b;c; Pennington et al., 2014; Mnih & Teh, 2012; Mnih & Kavukcuoglu, 2013; Levy & Goldberg, 2014; Vilnis & McCallum, 2015; Barkan, 2016; Bamler & Mandt, 2017). Most of thes”
A Cookbook of Self-Supervised Learning (2023, methodology): “This is an alternative to sampling based estimation of the gradient that was found to be less stable [Bengio and Senécal, 2003, 2008].”
A Batch Noise Contrastive Estimation Approach for Training Large Vocabulary Language Models (2017, background): “Unfortunately, this approach requires a control of the samples variance, which can lead otherwise to unstable learning [12].”
KNET: A General Framework for Learning Word Embedding Using Morphological Knowledge (2015, methodology): “Several approaches [Morin and Bengio 2005; Bengio et al. 2003; Bengio and Senecal 2008] have been employed to tackle this problem.”
Learning word embeddings efficiently with noise-contrastive estimation (2013, methodology): “Using importance sampling to approximate the likelihood gradient was the first such method to be proposed [2, 3], and though it could produce substantial speedups, it suffered from stability problems.”