Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation

By Yoshua Bengio, Nicholas Léonard, Aaron C. Courville. 2013

4,047 citations recorded by the source corpus.

Stochastic neurons and hard non-linearities can be useful for a number of reasons in deep learning models, but in many cases they pose a challenging problem: how to estimate the gradient of a loss function with respect to the input of such stochastic or non-smooth neurons? I.e., can we "back-propagate" through these stochastic neurons? We examine this question, existing approaches, and compare four families of solutions, applicable in different settings. One of them is the minimum variance unbiased gradient estimator for stochatic binary neurons (a special case of the REINFORCE algorithm). A second approach, introduced here, decomposes the operation of a binary stochastic neuron into a stochastic binary part and a smooth differentiable part, which approximates the expected effect of the pure stochatic binary neuron to first order. A third approach involves the injection of additive or multiplicative noise in a computational graph that is otherwise differentiable. A fourth approach heuristically copies the gradient with respect to the stochastic output directly as an estimator of the gradient with respect to the sigmoid argument (we call this the straight-through estimator). To explore a context where these estimators are useful, we consider a small-scale version of {\em conditional computation}, where sparse stochastic units form a distributed representation of gaters that can turn off in combinatorially many ways large chunks of the computation performed in the rest of the neural network. In this case, it is important that the gating units produce an actual 0 most of the time. The resulting sparsity can be potentially be exploited to greatly reduce the computational cost of large deep networks for which conditional computation would be useful.

Subfields: Artificial Intelligence, Statistics and Probability, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

In Model Architectures

Citations in the corpus

Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, background): “…forms of conditional computation have been proposed as a way to increase model capacity without a proportional increase in computational costs (Davis & Arel, 2013; Bengio et al., 2013; Eigen et al., 2013; Ludovic Denoyer, 2014; Cho & Bengio, 2014; Bengio et al., 2015; Almahairi et al., 2015).”
Deep AutoRegressive Networks (2013, background): “Whilst it is possible to derive estimators that are unbiased (Bengio et al., 2013a), their empirical performance is often unsatisfactory.”
Language Models are Few-Shot Learners (2020, methodology): “arameter count but not computation, as a means of increasing models’ capacity to store information without increased computational cost. These approaches rely on the conditional computation framework [BLC13] and speciﬁcally, the mixture-of-experts method [SMM+17] has been used to produce 100 billion”
Unified Scaling Laws for Routed Language Models (2022, methodology): “Finally, we re-analyze a technique that optimizes the router via Reinforcement Learning (a class of methods we call RL-R), which was proposed in early work on neural conditional computation [Bengio et al., 2013, 2016, Bengio, 2017, Denoyer and Gallinari, 2014].”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “To more efficiently increase model capacity, conditional compute strategies have been developed (Bengio et al., 2013; Davis and Arel, 2013; Cho and Bengio, 2014; Bengio et al., 2015), where each input activates a subset of the model.”
GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (2020, background): “In this section, we advocate how conditional computation (Bengio et al., 2013; Davis & Arel, 2013) with sparsely gated mixture of experts ﬁts into the above detailed desiderata and show its efﬁcacy by scaling neural machine translation models, while keeping the training time of such massive…”
DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “Our work is also related to the conditional computation models that activate parts of the neural network based on the input [2, 1, 33, 12, 36].”
Zero-Shot Text-to-Image Generation (2021, methodology): “Oord et al. (2017); Razavi et al. (2019) address this using an online cluster assignment procedure coupled with the straight-through estimator (Bengio et al., 2013).”
CogView: Mastering Text-to-Image Generation via Transformers (2021, methodology): “• The nearest-neighbor mapping, straight-through estimator [2], which is proposed by the original VQVAE.”
Taming Transformers for High-Resolution Image Synthesis (2020, background): “(3) is achieved by a straight-through gradient estimator, which simply copies the gradients from the decoder to the encoder [3], such that the model and codebook can be trained end-to-end via the loss function”
Gradient Estimation Using Stochastic Computation Graphs (2015, background): “[2], though they are concerned (3)The optimal baseline for scalar ✓ is in fact the weighted expectation E x [ f(x)s(x)2 ]”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “As in van den Oord et al. (2017), we use the straight-through estimator (Bengio et al., 2013) to compute the gradient passed from qt to xt.”
Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning (2019, background): “1) causes the quantization operation to be non-differentiable, the gradient of the encoder is approximated by straight-through (ST) gradient estimator [15].”
Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks (2019, methodology): “Thirdly, we consider different discretisation methods; we show that the categorical VAE (Cat-VAE) [30, 31] and straight-through estimation (STE) [27–29] both perform competitively to the VQ-VAE.”
Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, methodology): “During the backward pass, the gradient of the loss with respect to the pre-quantized embedding is approximated using the straight-through estimator [42], i.”
The challenge of realistic music generation: modelling raw audio at scale (2018, background): “The projection onto the simplex limits the maximal quantisation error, which makes the gradients that pass through it (using straight-through estimation [4]) more accurate.”
Categorical Reparameterization with Gumbel-Softmax (2016, methodology): “…τ is a learned parameter (rather than annealed via a fixed schedule), this scheme can be interpreted as entropy regularization (Szegedy et al., 2015; Pereyra et al., 2016), where the Gumbel-Softmax distribution can adaptively adjust the “confidence” of proposed samples during the training process.”