Stochastic Backpropagation through Mixture Density Distributions

By Alex Graves. 2016

48 citations recorded by the source corpus.

The ability to backpropagate stochastic gradients through continuous latent distributions has been crucial to the emergence of variational autoencoders and stochastic gradient variational Bayes. The key ingredient is an unbiased and low-variance way of estimating gradients with respect to distribution parameters from gradients evaluated at distribution samples. The "reparameterization trick" provides a class of transforms yielding such estimators for many continuous distributions, including the Gaussian and other members of the location-scale family. However the trick does not readily extend to mixture density models, due to the difficulty of reparameterizing the discrete distribution over mixture weights. This report describes an alternative transform, applicable to any continuous multivariate distribution with a differentiable density function from which samples can be drawn, and uses it to derive an unbiased estimator for mixture density weight derivatives. Combined with the reparameterization trick applied to the individual mixture components, this estimator makes it straightforward to train variational autoencoders with mixture-distributed latent variables, or to perform stochastic variational inference with a mixture density variational posterior.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

A Comprehensive guide to Bayesian Convolutional Neural Network with Variational Inference (2019, background): “Graves [20] derives a similar algorithm in the case of a mixture posterior probability distribution.”
Discrete Variational Autoencoders (2016, background): “Graves (2016) computes VAE-like gradient approximations for mixture models, but the component models are assumed to be simple factorial distributions.”
Uncertainty Estimations by Softplus normalization in Bayesian Convolutional Neural Networks with Variational Inference (2018, methodology): “[18] derives a similar algorithm in the case of a mixture posterior probability distribution.”
Bayesian Convolutional Neural Networks (2018, unknown): “Graves (2016) derives a similar algorithm in the case of a mixture posterior.”
Bayesian Recurrent Neural Networks (2017, background): “Graves (2016) derives a similar algorithm in the case of a mixture posterior.”
DVAE#: Discrete Variational Autoencoders with Relaxed Boltzmann Priors (2018, methodology): “Our approach is a special case of the reparameterization method for multivariate mixture distributions proposed in [42].”
Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoders (2016, methodology): “Graves (2016) also suggested an algorithm for backpropagation through GMMs.”
Bayesian Convolutional Neural Networks with Variational Inference (2018, background): “Graves [9] derives a similar algorithm in the case of a mixture posterior probability distribution.”
Monte Carlo Gradient Estimation in Machine Learning (2019, methodology): “Graves (2016) derives the implicit reparameterisation gradients for multivariate distributions with analytically tractable CDFs, such as mixtures.”
Doubly Reparameterized Gradient Estimators for Monte Carlo Objectives (2018, background): “Notably, the recently introduced implicit reparameterization trick has expanded the class of distributions for which we can compute reparameterization gradients to include Gaussian mixtures among many others (Graves, 2016; Jankowiak & Obermeyer, 2018; Jankowiak & Karaletsos, 2018; Figurnov et al., 2”