Stochastic Back-propagation and Variational Inference in Deep Latent Gaussian Models

By Danilo Jimenez Rezende, Shakir Mohamed, Daan Wierstra. 2014

145 citations recorded by the source corpus.

We marry ideas from deep neural networks and approximate Bayesian inference to derive a generalised class of deep, directed generative models, endowed with a new algorithm for scalable inference and learning. Our algorithm introduces a recognition model to represent approximate posterior distributions, and that acts as a stochastic encoder of the data. We develop stochastic back-propagation -- rules for back-propagation through stochastic variables -- and use this to develop an algorithm that allows for joint optimisation of the parameters of both the generative and recognition model. We demonstrate on several real-world data sets that the model generates realistic samples, provides accurate imputations of missing data and is a useful tool for high-dimensional data visualisation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “Many of these approaches rely either on auto-encoding of images [24, 25, 26] or on adversarial learning [27], jointly modelling data and representation [28, 29, 30, 31].”
Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, background): “In [19, 10], followed by [31, 30, 28], variational learning and inference algorithms were developed which allow a flexible generative model and posterior distribution over latent variables to be directly trained against each other.”
Unrolled Generative Adversarial Networks (2016, background): “…training losses and model architectures, including denoising autoencoders (Vincent et al., 2010), variational autoencoders (Kingma & Welling, 2013; Rezende et al., 2014; Gregor et al., 2015; Kulkarni et al., 2015; Burda et al., 2015; Kingma et al., 2016), generative stochastic networks (Alain et…”
Unsupervised Image-to-Image Translation Networks (2017, methodology): “The framework, as illustrated in Figure 1, is motivated by recent deep generative models including variational autoencoders (VAEs) (Kingma & Welling, 2013; Rezende et al., 2014; Larsen et al., 2016) and generative adversarial networks (GANs) (Goodfellow et al.”
Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networks (2015, background): “Variational auto-encoders [16, 24] provide probabilistic interpretation which facilitates sampling.”
Generating Images from Captions with Attention (2015, background): “Kingma & Welling (2014), Rezende et al. (2014) have introduced the Variational Auto-Encoder (VAE) which can be seen as a neural network with continuous latent variables.”
Scalable Bayesian Optimization Using Deep Neural Networks (2015, methodology): “Bayesian optimization proceeds by performing a proxy optimization over this acquisition function in order to determine the next input to evaluate.”
Variational Information Maximisation for Intrinsically Motivated Reinforcement Learning (2015, background): “Other lower bounds for the mutual information are also possible: Jaakkola & Jordan [9] present a lower bound by using the convexity bound for the logarithm; Brunel & Nadal [2] use a Gaussian assumption and appeal to the Cramer-Rao lower bound.”
Hierarchical Variational Models (2015, methodology): “The use of inference networks (Dayan, 2000; Stuhlmüller et al., 2013; Kingma and Welling, 2014; Rezende et al., 2014) amortizes the cost of estimating these local variational parameters by tying them together through a neural network.”
MoCoGAN: Decomposing Motion and Content for Video Generation (2017, background): “Various deep generative models were recently proposed for image generation including GANs [12], variational autoencoders (VAEs) [20, 29, 37], and PixelCNNs [39].”
Techniques for Learning Binary Stochastic Feedforward Neural Networks (2014, background): “It would be an interesting line of research to train an auxiliary model for the proposal distribution R(h|x,y) following ideas from (Kingma and Welling, 2013; Rezende et al., 2014; Mnih and Gregor, 2014) that call the equivalent of R the recognition model or the inference network.”
Learning Stochastic Recurrent Networks (2014, methodology): “Using stochastic gradient variational Bayes ( SGVB ) [17, 18] as an estimator, we train RNNs to model high dimensional sequences.”
Improved generator objectives for GANs (2016, background): “The visual quality of samples from generative models trained with GANs often exceeds those of their variationally-trained counterparts [5, 12].”
Generative Neurosymbolic Machines (2020, background): “Deep latent variable models like variational autoencoders (VAEs) [28, 36] offer an elegant probabilistic framework to learn both these abilities in an unsupervised and end-to-end trainable fashion.”
Maximizing acquisition functions for Bayesian optimization (2018, methodology): “Roughly speaking, a set function L is SM if its increase in value when adding any new point x j to an existing collection X j is non-increasing in cardinality k (for a technical overview, see [2]).”