REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models

By George Tucker, Andriy Mnih, Chris J. Maddison, John Lawson, Jascha Narain Sohl-Dickstein. 2017

303 citations recorded by the source corpus.

Learning in models with discrete latent variables is challenging due to high variance gradient estimators. Generally, approaches have relied on control variates to reduce the variance of the REINFORCE estimator. Recent work \citep{jang2016categorical, maddison2016concrete} has taken a different approach, introducing a continuous relaxation of discrete variables to produce low-variance, but biased, gradient estimates. In this work, we combine the two approaches through a novel control variate that produces low-variance, \emph{unbiased} gradient estimates. Then, we introduce a modification to the continuous relaxation and show that the tightness of the relaxation can be adapted online, removing it as a hyperparameter. We show state-of-the-art variance reduction on several benchmark generative modeling tasks, generally leading to faster convergence to a better final log-likelihood.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Optimization & Training

Citations in the corpus

MaskGAN: Better Text Generation via Filling in the ______ (2018, background): “Recent unbiased and low variance gradient estimate techniques such as Tucker et al. (2017) may prove more effective.”
NVAE: A Deep Hierarchical Variational Autoencoder (2020, background): “…as reducing the gap between approximate and true posterior distributions [3, 4, 5, 6, 7, 8, 9, 10], formulating tighter bounds [11, 12, 13, 14], reducing the gradient noise [15, 16], extending VAEs to discrete variables [17, 18, 19, 20, 21, 22, 23], or tackling posterior collapse [24, 25, 26, 27].”
Foundation Posteriors for Approximate Probabilistic Inference (2022, neutral): “For discrete random variables, future work can study using the concrete relaxation [42] or the family of REINFORCE-based techniques [65, 46, 60].”
Learning Sparse Neural Networks through L0 Regularization (2017, background): “Nevertheless, it should be mentioned that there is encouraging work on improving upon the high variance gradients of REINFORCE with control variates (Mnih & Gregor, 2014; Mnih & Rezende, 2016; Tucker et al., 2017), which however usually requires multiple evaluations of the model.”
Automatic differentiation in machine learning: a survey (2015, neutral): “For discrete variables, the REBAR (Tucker et al., 2017) method”
Backpropagation through the Void: Optimizing control variates for black-box gradient estimation (2017, methodology): “We generalize the method of Tucker et al. (2017) to learn a free-form control variate parameterized by a neural network.”
Routing Networks and the Challenges of Modular and Compositional Computation (2019, background): “The first, REBAR [Tucker et al., 2017] compensates for REINFORCE’s variance by introducing a control variate based on the concrete distribution.”
Fast Decoding in Sequence Models using Discrete Latent Variables (2018, methodology): “Recent follow up work to the VIMCO includes REBAR (Tucker et al., 2017) which was further improved in RELAX (Grathwohl et al.”
DVAE++: Discrete Variational Autoencoders with Overlapping Transformations (2018, methodology): “We use the settings in (Tucker et al., 2017) to initialize the parameters, deﬁne the model, and optimize the parameters for the same number of iterations.”
A RAD approach to deep mixture models (2019, background): “Similar to Rolfe (2017), the learning of discrete variables is achieved by relying on the the continuous component of the model, and, as opposed to other approaches (Bengio et al., 2013; Raiko et al., 2015; Jang et al., 2017; Maddison et al., 2017; Grathwohl et al., 2018; Tucker et al., 2017; Tran e”
DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors (2018, methodology): “REBAR [37] and its generalization RELAX [38] use the reparameterization of continuous distributions to deﬁne baselines.”
Monte Carlo Gradient Estimation in Machine Learning (2019, background): “Coverage of this discrete distribution setting makes for an interesting review paper in itself, and existing work includes Glasserman and Ho (1991); Gu et al. (2016); Maddison et al. (2016); Tucker et al. (2017).”
Adversarial Contrastive Estimation (2018, methodology): “More advanced gradient estimators exist that also reduce variance (Grathwohl et al., 2017; Tucker et al., 2017; Liu et al., 2018), but for simplicity we use the self-critical baseline method (Rennie et al., 2016), where the baseline is b ( x ) = l ω ( y + , y (cid:63) , x ) , or b ( x ) = − ˜ s ω (…”
DiCE: The Infinitely Differentiable Monte-Carlo Estimator (2018, methodology): “In particular we hope to include solutions such as REBAR (Tucker et al., 2017) in the DICE operator.”
Latent Alignment and Variational Attention (2018, methodology): “…easily computable, but there is an optimization issue with the gradient with respect to the ﬁrst term Many recent methods target this issue, including neural estimates of baselines [50, 51], Rao-Blackwellization [59], reparameterizable relaxations [31, 47], and a mix of various techniques [73, 24].”
Learning in the Frequency Domain (2020, methodology): “[33, 34, 35] propose a reparameterization method, called Gumbel Softmax trick, which allows the gradients to back propagate through a discrete sampling process (see Gumbel samples in Figure 4).”
Theory and Experiments on Vector Quantized Autoencoders (2018, background): “There have been several attempts to reduce the variance in the gradients [20, 21, 32, 6].”
Gradient Estimation with Stochastic Softmax Tricks (2020, background): “Building on the score function estimator are a variety of estimators that require multiple evaluations of L to reduce variance [32, 79, 29, 85, 44, 9].”