Estimating or Propagating Gradients Through Stochastic Neurons

By Yoshua Bengio. 2013

132 citations recorded by the source corpus.

Stochastic neurons can be useful for a number of reasons in deep learning models, but in many cases they pose a challenging problem: how to estimate the gradient of a loss function with respect to the input of such s tochastic neurons, i.e., can we “back-propagate” through these stochastic neurons? We examine this question, existing approaches, and present two novel families of solutions, applicable in different settings. In particular, it is demonstrate d that a simple biologically plausible formula gives rise to an an unbiased (but noisy) estimator of the gradient with respect to a binary stochastic neuron firing proba bility. Unlike other estimators which view the noise as a small perturbation in order to estimate gradients by finite differences, this estimator is unbiased even w ithout assuming that the stochastic perturbation is small. This estimator is also in teresting because it can be applied in very general settings which do not allow gradient back-propagation, including the estimation of the gradient with respect to futur e rewards, as required in reinforcement learning setups. We also propose an approach to approximating this unbiased but high-variance estimator by learning to predict it using a biased estimator. The second approach we propose assumes that an estimator of the gradient can be back-propagated and it provides an unbiased estimator of the gradient, but can only work with non-linearities unlike the hard threshold, but like the rectifier, that are not flat for all of their range. This is similar to trad itional sigmoidal units but has the advantage that for many inputs, a hard decision (e.g., a 0 output) can be produced, which would be convenient for conditional computation and achieving sparse representations and sparse gradients.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Deep Generative Stochastic Networks Trainable by Backprop (2013, background): “Those conclusions show that our Markov chain has the property that its samples in X are drawn from the same distribution as X 0 .”
GSNs : Generative Stochastic Networks (2015, background): “An interesting source of inspiration regarding this question is a recent paper on estimating or propagating gradients through stochastic neurons (Bengio, 2013).”
Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activations (2016, background): “Bengio (2013) studied the question of estimating or propagating gradients through stochastic discrete neurons.”
Getting a CLUE: A Method for Explaining Uncertainty Estimates (2020, neutral): “This is known as the softmax straight-through estimator [45].”
Deep Neural Decision Trees (2018, methodology): “mbel-Softmax (Jang et al.,2017) can be applied: for the forward pass, we sample a one-hot vector using Gumbel-Max trick, while for the backward pass, we use Gumbel-Softmax to compute the gradient (seeBengio (2013) for a more detailed analysis). Fig.1demonstrates a concrete example where we have a sc”
Variational Dropout and the Local Reparameterization Trick (2015, background): “Variable reparameterizations have a long history in the statistics literature, but have only recently found use for efficient gradient-based machine learning and inference [4, 13, 19].”
Techniques for Learning Binary Stochastic Feedforward Neural Networks (2014, background): “More recent work proposes simply drawing samples from P ( h | x ) during the feedforward phase (Hinton et al. , 2012; Bengio, 2013; Tang and Salakhutdinov, 2013).”
Deep Learning of Representations: Looking Forward (2013, methodology): “A Taylor expansion of the denoising error shows that it is also approximately equivalent to minimizing reconstruction error plus a contractive penalty on r(·) (Alain and Bengio, 2013).”
Deep Directed Generative Autoencoders (2014, background): “A similar question was raised in a different context in Bengio (2013); Bengio et al. (2013b), and a number of possible update directions were compared.”
Understanding Uncertainty in Bayesian Neural Networks (2019, unknown): “This is known as the softmax straight-through estimator (Bengio, 2013).”
Noisy Activation Functions (2016, methodology): “This is different from previous proposals such as the noisy rectifier from Bengio (2013) where noise is added just before a rectifier (ReLU) unit, independently of whether the input is in the linear regime or in the saturating regime of the nonlinearity.”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “Conditional computing applies to training and inference [20, 53].”
Mollifying Networks (2016, methodology): “The task is to predict whether if there is a different shape in the image or not (Gülçehre and Bengio, 2013).”
Bounding the Test Log-Likelihood of Generative Models (2013, background): “…each model: GSN-1 (DAE): • Architecture: 784 (input) - 2000 (tanh) • Noise: (input) 0.28 salt-and-pepper, (hidden) no noise • Learning: 9-step walkback (Bengio, 2013), learning rate 0.05, cross-entropy cost, 200 epochs • Early-stop: visual inspection of generated samples GSN-2: • Architecture:…”
Difference Target Propagation (2014, background): “Recently, stochastic networks have attracted attention [3, 20, 5] because they are able to learn a multi-modal conditional distribution”
Latent Tree Learning with Differentiable Parsers: Shift-Reduce Parsing and Chart Parsing (2018, methodology): “Hence for fairness, in our experiments we replace the softmax-with-temperature of Maillard et al. (2017) with a softmax followed by a straight-through estimator (Bengio et al., 2013).”