MuProp: Unbiased Backpropagation for Stochastic Neural Networks

By Shixiang Shane Gu, Sergey Levine, Ilya Sutskever, Andriy Mnih. 2015

147 citations recorded by the source corpus.

Deep neural networks are powerful parametric models that can be trained efficiently using the backpropagation algorithm. Stochastic neural networks combine the power of large parametric functions with that of graphical models, which makes it possible to learn very complex distributions. However, as backpropagation is not directly applicable to stochastic networks that include discrete sampling operations within their computational graph, training such networks remains difficult. We present MuProp, an unbiased gradient estimator for stochastic networks, designed to make this task easier. MuProp improves on the likelihood-ratio estimator by reducing its variance using a control variate based on the first-order Taylor expansion of a mean-field network. Crucially, unlike prior attempts at using backpropagation for training stochastic networks, the resulting estimator is unbiased and well behaved. Our experiments on structured output prediction and discrete latent variable modeling demonstrate that MuProp yields consistently good performance across a range of difficult tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Categorical Reparameterization with Gumbel-Softmax (2016, methodology): “This is a common benchmark for training stochastic binary networks (SBN) (Raiko et al., 2014; Gu et al., 2016; Mnih & Rezende, 2016).”
Data-efficient Deep Reinforcement Learning for Dexterous Manipulation (2017, background): “…there has been a resurgence of interest in end-to-end training of neural networks for challenging control problems, and several algorithms, both value and policy focused have been developed and applied to challenging problems including continuous control, e.g. [22, 23, 6, 7, 20, 32, 31, 11, 17].”
Q-Prop: Sample-Efficient Policy Gradient with An Off-Policy Critic (2016, methodology): “Off-policy methods, such as Q-learning (Watkins & Dayan, 1992; Sutton et al., 1999; Mnih et al., 2015; Gu et al., 2016b) and off-policy actor-critic methods (Lever, 2014; Lillicrap et al., 2016), can instead use all samples, including off-policy samples, by adopting temporal difference learning…”
The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables (2016, methodology): “For models with discrete latent variables, the discontinuous nature of which makes reparameterization not useful, a number of score function estimators have been developed [31, 17, 33, 28, 42, 18], which differ primarily in the variance reduction techniques used.”
SNAS: Stochastic Neural Architecture Search (2018, background): “Actually, the idea to make use of this gradient information to improve the learning efﬁciency of a stochastic model has been discussed in the literature of generative model (Gu et al., 2015; Maddison et al., 2016) and reinforcement learning (Schmidhuber, 1990; Arjona-Medina et al., 2018).”
Boundary-Seeking Generative Adversarial Networks (2017, background): “Approximations for the back-propagated signal exist (Bengio et al., 2013; Gu et al., 2015; Gumbel & Lieblein, 1954; Jang et al., 2016), and we discuss comparisons in more detail in the Results section.”
DVAE++: Discrete Variational Autoencoders with Overlapping Transformations (2018, methodology): “MuProp (Gu et al., 2015) and DARN (Gre-gor et al., 2013) are two other REINFORCE-based methods (with non-zero biases) that use a Taylor expansion of the function f ( zzz ) to create control variates.”
The Generalized Reparameterization Gradient (2016, methodology): “Ranganath et al. (2014) reduce the variance of these estimates using Rao-Blackwellization (Casella and Robert, 1996) and control variates (Ross, 2002; Paisley et al., 2012; Gu et al., 2016).”
REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models (2017, methodology): “Across tasks, we show that REBAR has state-of-the-art variance reduction which translates to faster convergence and better final log likelihoods.”
Undirected Graphical Models as Approximate Posteriors (2019, methodology): “…apply to discrete latent variables and recent methods for discrete variables have focused on REINFORCE with control variates (Mnih & Gregor, 2014; Gu et al., 2015; Mnih & Rezende, 2016; Tucker et al., 2017; Grathwohl et al., 2018) or continuous relaxations (Maddison et al., 2016; Jang et al.,…”
DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors (2018, methodology): “NVIL [3] uses an input-dependent baseline, and MuProp [35] uses a ﬁrst-order Taylor approximation along with an input-dependent baseline to reduce noise.”
Direct Optimization through arg max for Discrete Variational Auto-Encoder (2019, methodology): “While one can use unbiased estimators, such as REINFORCE [38], their variance is typically high [26, 23, 35, 5, 24, 36].”
An Introduction to Variational Autoencoders (2019, methodology): “These proposals include NVIL (Mnih and Gregor, 2014), MuProp (Gu et al., 2015), Variational inference for Monte Carlo objectives (Mnih and Rezende, 2016), the Concrete distribution (Maddison et al.”
Gradient Estimation with Stochastic Softmax Tricks (2020, background): “[32] Shixiang Gu, Sergey Levine, Ilya Sutskever, and Andriy Mnih.”
Variational Rejection Sampling (2018, methodology): “…particular, Wingate and Weber [2013] and Ranganath et al. [2014] use simple scalar CV, NVIL proposed input-dependent CV [Mnih and Gre-gor, 2014], and MuProp combines input-dependent CV with deterministic ﬁrst-order Taylor approximations to the mean-ﬁeld approximation of the model [Gu et al., 2016].”
Masked Gradient-Based Causal Structure Learning (2019, methodology): “, 2013) and several REINFORCE based methods (Mnih & Gregor, 2014; Gu et al., 2016).”
Bayesian Prediction of Future Street Scenes using Synthetic Likelihoods (2018, background): “Although there has been significant effort in improving training efficiency (Rezende et al., 2014; Gu et al., 2016), success has been partial.”