Quick Training of Probabilistic Neural Nets by Importance Sampling

By Yoshua Bengio, Jean-Sébastien Senecal. 2003

35 citations recorded by the source corpus.

Subfields: Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

A fast and simple algorithm for training neural probabilistic language models (2012, methodology): “Perhaps a more elegant approach is to keep the model the same and to approximate the expensive gradient computations using importance sampling (Bengio & Sen´ecal, 2003).”
The representational geometry of word meanings acquired by neural machine translation models (2017, methodology): “The algorithm uses (biased) importance sampling (Bengio and Sénécal 2003) to approximate the probability distribution of words over a large target vocabulary with a finite set of distributions over subsets of that vocabulary.”
Notes on Noise Contrastive Estimation and Negative Sampling (2014, methodology): “Since the derivatives of the log likelihood include terms that are the expected values of the parameters under the model distribution, the classic strategy has been to use importance sampling and related Monte Carlo techniques to approximate these expectations (Bengio and Sen´ecal, 2003).”
Modeling Vocabulary for Big Code Machine Learning (2019, unknown): “edicting a class, before predicting a word among a subset [27]. Morin and Bengio adapt this approach to Neural LMs [49]. Bengio and Senécal propose the use of importance sampling to train a Neural LM [14], although this approach provides a speedup only during training, not inference. Importance samp”
Top-K Off-Policy Correction for a REINFORCE Recommender System (2018, methodology): “on space. The normalization term in the softmax requires going over all the possible actions, which is in the order of millions in our setting. To speed up the computation, we perform sampled softmax [4] during training. At serving time, we used an efficient nearest neighbor search algorithm to retr”