Backpropagation through the Void: Optimizing control variates for black-box gradient estimation

By Will Grathwohl, Dami Choi, Yuhuai Wu, Geoffrey Roeder, David Kristjanson Duvenaud. 2017

317 citations recorded by the source corpus.

Gradient-based optimization is the foundation of deep learning and reinforcement learning. Even when the mechanism being optimized is unknown or not differentiable, optimization using high-variance or biased gradient estimates is still often the best strategy. We introduce a general framework for learning low-variance, unbiased gradient estimators for black-box functions of random variables. Our method uses gradients of a neural network trained jointly with model parameters or policies, and is applicable in both discrete and continuous settings. We demonstrate this framework for training discrete latent-variable models. We also give an unbiased, action-conditional extension of the advantage actor-critic reinforcement learning algorithm.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Optimization & Training

Citations in the corpus

NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “…as reducing the gap between approximate and true posterior distributions [3, 4, 5, 6, 7, 8, 9, 10], formulating tighter bounds [11, 12, 13, 14], reducing the gradient noise [15, 16], extending VAEs to discrete variables [17, 18, 19, 20, 21, 22, 23], or tackling posterior collapse [24, 25, 26, 27].”
DADA: Differentiable Automatic Data Augmentation (2020, positive): “To achieve accurate gradient estimation, we introduce an unbiased gradient estimator, RELAX [7], to our DADA framework.”
The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, methodology): “Several recent methods (Gu et al., 2017a; Thomas & Brunskill, 2017; Grathwohl et al., 2018; Liu et al., 2018; Wu et al., 2018) have extended the approach to state-action-dependent baselines (i.”
From Importance Sampling to Doubly Robust Policy Gradient (2019, critical): “More recently, Cheng et al. (2019) noticed that the previous variance reduction methods in PG overlooked the correlation across the times steps and ignored the randomness in the future steps (e.g., Grathwohl et al., 2018; Gu et al., 2017; Liu et al., 2018; Wu et al., 2018).”
Reward-estimation variance elimination in sequential decision processes (2018, background): “More recently state-action-dependent baselines have been investigated [2, 4, 8, 20].”
Automatic differentiation in machine learning: a survey (2015, methodology): “A generalization of REBAR called RELAX (Grathwohl et al., 2017) works by learning a freeform control variate parameterized by a neural network and is applicable in both discrete and continuous settings.”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, background): “As with other experiments, the (Q-)value based approaches start learning, but, more surprisingly, RELAX is able to stabilize, although at the cost of complete collapse.”
Fast Decoding in Sequence Models using Discrete Latent Variables (2018, background): “, 2017) which was further improved in RELAX (Grathwohl et al., 2017).”
Generalizable Adversarial Attacks with Latent Variable Perturbation Modelling (2019, unknown): “to xis only feasible if the input domain Xis a 2Our framework could be extended to the targeted setting by modifying the loss function, as well as to blackbox attacks via blackbox gradient estimation [34, 11], but we leave these extensions to future work. 2 Figure 1: The main components of DAGAER an”
Reparameterization Gradient for Non-differentiable Models (2018, background): “Also, a variety of control variates for the standard score estimator [35, 27, 36, 28] for the gradients of variational objectives have been developed [28, 7, 8, 34, 6, 23], some of which use biased yet differentiable control variates such that the reparameterization trick can be used to correct the ”
Adversarial Variational Optimization of Non-Differentiable Simulators (2017, unknown): “, 2011) or variance reduction techniques such as control variates (Grathwohl et al., 2017) or finite differences (Buesing et al.”
Undirected Graphical Models as Approximate Posteriors (2019, methodology): “Reparameterization does not apply to discrete latent variables and recent methods for discrete variables have focused on REINFORCE with control variates (Mnih & Gregor, 2014; Gu et al., 2015; Mnih & Rezende, 2016; Tucker et al., 2017; Grathwohl et al., 2018) or continuous relaxations (Maddison et al”
Gradient Estimation with Stochastic Softmax Tricks (2020, background): “Building on the score function estimator are a variety of estimators that require multiple evaluations of L to reduce variance [32, 79, 29, 85, 44, 9].”
The Thermodynamic Variational Objective (2019, methodology): “…is used to evaluate objectives, continuous relaxations and control variate methods for learning discrete latent variable models (Mnih and Gregor, 2014; Maddison et al., 2017b; Jang et al., 2017; Mnih and Rezende, 2016; Bornschein and Bengio, 2015; Tucker et al., 2017; Grathwohl et al., 2018).”
A RAD approach to deep mixture models (2019, background): “Similar to Rolfe (2017), the learning of discrete variables is achieved by relying on the the continuous component of the model, and, as opposed to other approaches (Bengio et al., 2013; Raiko et al., 2015; Jang et al., 2017; Maddison et al., 2017; Grathwohl et al., 2018; Tucker et al., 2017; Tran e”
Action-depedent Control Variates for Policy Optimization via Stein's Identity (2017, background): “Such works include Grathwohl et al. (2018) which train an action-dependent baseline for both discrete and continuous control tasks.”
Direct Optimization through arg max for Discrete Variational Auto-Encoder (2019, methodology): “We also compare the unbiased estimators REBAR [33] and RELAX [7] and the recent ARM estimator [38].”
Deep Reinforcement Learning (2018, methodology): “, 2016), control-variates optimization (Grathwohl et al., 2017), updating the target policy via the expectation of its actions (Ciosek & Whiteson, 2018; Asadi et al.”