Quick Training of Probabilistic Neural Nets by Importance Sampling

By Blockin Blockin. 2003

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Exploring the Limits of Language Modeling (2016, background): “Several approaches have been proposed to cope with the scaling issue: importance sampling (Bengio et al., 2003; Bengio & Senécal, 2008), Negative Contrast Estimation (NCE) (Gutmann & Hyvärinen, 2010; Mnih & Kavukcuoglu, 2013), self normalizing partition functions (Vincent et al.”
Learning Visual N-Grams from Web Data (2016, methodology): “To circumvent this issue, we follow [28] and perform stochastic gradient descent over outputs [4]: we only perform the forwardbackward pass for a random subset (formed by all positive n-grams in the batch) of the columns of E.”
Efficient softmax approximation for GPUs (2016, methodology): “Our method is compared to: (1) the full softmax, (2) the hierarchical softmax (HSM) with frequency binning [31], (3) importance sampling [5, 4] and (4) the differentiated softmax [8].”
A Neural Knowledge Language Model (2016, methodology): “These methods approximate the softmax output using hierarchical softmax [31, 29], importance sampling [5, 22], noise contrastive estimation [30], etc.”
Augmenting Self-attention with Persistent Memory (2019, methodology): “Solutions are to either replace the softmax loss with an approximation [12, 29], to sample from the vocabulary during training [5, 21] or to include subword units [34].”
BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies (2015, background): “In the past, a uniform distribution or the unigram distribution have been advocated as promising candidates for sampling distributions (Bengio & Sen ´ ecal, 2003; Jean et al., 2015; Bengio & Sen ´ ecal, 2008; Mnih & Teh, 2012).”
Continuous space language models (2007, methodology): “Therefore several speed-up techniques have been proposed in the literature, in particular parallel implementations (Bengio et al., 2003), resampling techniques (Bengio and Sénécal, 2003) and hierarchical decomposition (Morin and Bengio, 2005).”
On Sampled Metrics for Item Recommendation (2020, background): “Sampling of negatives is commonly used during training of large models [4, 13], and several works have studied the implications of sampling as well as various methods to improve it [5, 6], see [18] for a comparative study of sampling methods.”
Learning word embeddings efficiently with noise-contrastive estimation (2013, methodology): “Using importance sampling to approximate the likelihood gradient was the first such method to be proposed [2, 3], and though it could produce substantial speedups, it suffered from stability problems.”
Exponential Family Embeddings (2016, methodology): “A neural language model (Bengio et al., 2003) uses neural networks to parameterize the n-gram probabilities of”
Online normalizer calculation for softmax (2018, methodology): “These performance improvements could be applied not only to the classical Softmax function; They are orthogonal to many other Softmax optimization techniques including Hierarchical Softmax, Importance Sampling, and SVD-Softmax.”
Sampling-bias-corrected neural modeling for large corpus item recommendations (2019, unknown): “Importance sampling and the corresponding bias reduction have been studied in MLP model [4, 5].”
Sentiment Embeddings with Applications to Sentiment Analysis (2016, unknown): “We propose learning sentiment embeddings that encode sentiment of texts in continuous word representation.”
Modeling Vocabulary for Big Code Machine Learning (2019, background): “Bengio and Senécal propose the use of importance sampling to train a Neural LM [14], although this approach provides a speedup only during training, not inference.”