Variational Inference for Monte Carlo Objectives

By Andriy Mnih, Danilo Jimenez Rezende. 2016

312 citations recorded by the source corpus.

Recent progress in deep latent variable models has largely been driven by the development of flexible and scalable variational inference methods. Variational training of this type involves maximizing a lower bound on the log-likelihood, using samples from the variational posterior to compute the required gradients. Recently, Burda et al. (2016) have derived a tighter lower bound using a multi-sample importance sampling estimate of the likelihood and showed that optimizing it yields models that use more of their capacity and achieve higher likelihoods. This development showed the importance of such multi-sample objectives and explained the success of several related approaches. We extend the multi-sample approach to discrete latent variables and analyze the difficulty encountered when estimating the gradients involved. We then develop the first unbiased gradient estimator designed for importance-sampled objectives and evaluate it at training generative and structured output prediction models. The resulting estimator, which is based on low-variance per-sample learning signals, is both simpler and more effective than the NVIL estimator proposed for the single-sample variational objective, and is competitive with the currently used biased estimators.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Neural Discrete Representation Learning (2017, methodology): “…with continuous features have been the focus of many previous work [16, 39, 6, 9] however we concentrate on discrete representations [27, 33, 8, 28] which are potentially a more natural ﬁt for many of the modalities we are interested in. Language is inherently discrete, similarly speech is…”
VQ-DRAW: A Sequential Discrete VAE (2020, background): “Other works do away with the reparameterization trick entirely, opting for other strategies for reducing variance in the variational lower bound [5] [6] [7].”
Energy-Inspired Models: Learning with Sampler-Induced Distributions (2019, background): “Initially, we thought that an unbiased, low-variance estimator could be constructed similarly to VIMCO [49], however, this estimator still had high variance.”
The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables (2016, methodology): “Concrete relaxations more uniformly outperformed VIMCO in this instance.”
Sequential Attend, Infer, Repeat: Generative Modelling of Moving Objects (2018, methodology): “We use the VIMCO gradient estimator of Mnih and Rezende, 2016 to backpropagate through the discrete latent variables z pres , and use reparameterisation for the continuous ones (Kingma and Welling, 2013).”
Disentangling by Factorising (2018, background): “…factors would require using discrete latent variables instead of Gaussians, but jointly modelling discrete and continuous factors of variation is a non-trivial problem that needs further research. ance reduction techniques for gradient estimation (Mnih & Rezende, 2016; Maddison et al., 2017).”
Unsupervised Learning of 3D Structure from Images (2016, methodology): “This requires either a differentiable renderer [19], or resort to gradient estimation techniques such as finite-differences [5] or Monte Carlo estimators [21, 1].”
PIPPS: Flexible Model-Based Policy Search Robust to the Curse of Chaos (2019, methodology): “When estimating the gradient from a batch, one can estimate leave-one-out baselines for each point to obtain an unbiased gradient estimator (Mnih & Rezende, 2016), i.e. b i = (cid:54) = i φ ( x j ) / ( P − 1) .”
Recent Advances in Autoencoder-Based Representation Learning (2018, background): “Many other works explore learning (variational) autoencoders with (vector-)quantized latent representation with a focus on generative modeling [62, 63, 59, 60] and compression [64], rather than representation learning.”
Discrete Variational Autoencoders (2016, methodology): “…baseline technique, originally called REINFORCE in the reinforcement learning literature (Mnih & Gregor, 2014; Williams, 1992; Bengio et al., 2013; Mnih & Rezende, 2016): where B ( x ) is a (possibly input-dependent) baseline, which does not affect the gradient, but can reduce the variance of a…”
Backpropagation through the Void: Optimizing control variates for black-box gradient estimation (2017, background): “NVIL (Mnih & Gregor, 2014) and VIMCO (Mnih & Rezende, 2016) provide reduced variance gradient estimation in the special case of discrete latent variable models and discrete latent variable models with Monte-Carlo objectives.”
Fast Decoding in Sequence Models using Discrete Latent Variables (2018, methodology): “The NVIL estimator of (Mnih & Gregor, 2014) proposes using a single sample objective to optimize the variational lower bound, while VIMCO (Mnih & Rezende, 2016) proposes us-ing a muliti-sample objective of (Burda et al., 2015) which further speeds up convergence by using multiple samples from the…”
Neural Scene De-rendering (2017, background): “learning problem, and adopt a multi-sample REINFORCE paradigm [23, 32] to address this issue.”
Discrete Sequential Prediction of Continuous Actions for Deep RL (2017, methodology): “To reduce variance, we employ a Monte Carlo baseline, G, which is the mean reward from K samples from π [23].”
DVAE++: Discrete Variational Autoencoders with Overlapping Transformations (2018, background): “VIMCO (Mnih & Rezende, 2016) is a multi-sample version of NVIL that has baselines tailored for each sample based on all the other samples.”
REBAR: Low-variance, unbiased gradient estimates for discrete latent variable models (2017, methodology): “It would be natural to explore the extension to the multi-sample case (e.g., VIMCO (Mnih & Rezende, 2016)), to leverage the layered structure in our models using Q -functions, and to apply this approach to reinforcement learning.”
Variational Memory Addressing in Generative Models (2017, methodology): “Here we show how to use VIMCO [21] to estimate the gradients for this model.”