Stochastic Backpropagation and Approximate Inference in Deep Generative Models

By Danilo Jimenez Rezende, Shakir Mohamed, Daan Wierstra. 2014

5,702 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

Citations in the corpus

Generative Adversarial Nets (2014, background): “[23] had developed more general stochastic backpropagation rules, allowing one to backpropagate through Gaussian distributions with finite variance, and to backpropagate to the covariance parameter as well as the mean.”
Neural Variational Inference and Learning in Belief Networks (2014, methodology): “Recently a method for training nonlinear models with continuous latent variables, called Stochastic Gradient Variational Bayes (SGVB), has been proposed by Kingma & Welling (2013) and Rezende et al. (2014).”
Deep Generative Stochastic Networks Trainable by Backprop (2013, background): “The advantage of that formulation is that one can directly back-propagated the reconstruction log-likelihood log P ( X 1 = x 0 | H 1 = f ( X 0 , Z 0 , H 0 )) into all the parameters of f and g (a similar idea was independently proposed in (Kingma, 2013) and also exploited in (Rezende et al., 2014)).”
Denoising Diffusion Probabilistic Models (2020, background): “While diffusion models might resemble flows [9, 46, 10, 32, 5, 16, 23] and VAEs [33, 47, 37], diffusion models are designed so that q has no parameters and the top-level latent xT has nearly zero mutual information with the data x0.”
High-Resolution Image Synthesis with Latent Diffusion Models (2021, background): “, imposes a slight KL-penalty towards a standard normal on the learned latent, similar to a VAE [45, 67], whereas VQ-reg.”
Neural Autoregressive Distribution Estimation (2016, background): “Kingma and Welling (2014); Rezende et al. (2014) proposed the variational autoencoder (VAE), where the model is the same as the Helmholtz machine, but with real-valued (usually Gaussian) stochastic units.”
Pixel Recurrent Neural Networks (2016, background): “Most work focuses on stochastic latent variable models such as VAE’s (Rezende et al., 2014; Kingma & Welling, 2013) that aim to extract meaningful representations, but often come with an intractable inference step that can hinder their performance.”
The State of Sparsity in Deep Neural Networks (2019, methodology): “For each training step, we sample weights from this distribution and use the reparameterization trick (Kingma & Welling, 2013; Rezende et al., 2014) to differentiate the loss w.r.t. the parameters through the sampling operation.”
Data-Efficient Image Recognition with Contrastive Predictive Coding (2019, background): “ntations that resemble simple cells in the brain [48]. The idea was later formalized in the context of probabilistic generative models [27], which have become a staple of modern unsupervised learning [28,22,35,56]. A key problem that generative models must overcome, however, is that not all low-leve”
Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks (2016, methodology): “Next, we test the effect of weight normalization applied to deep convolutional variational auto-encoders (CVAEs) [13, 24, 25], trained on the MNIST data set of images of handwritten digits and the CIFAR-10 data set of small natural images.”
Jukebox: A Generative Model for Music (2020, background): “Variational autoencoders (VAEs) (Rezende et al., 2014; Kingma & Welling, 2014) impose a Gaussian prior on the latent code in an encoder-decoder setup from which data can be sampled.”
Zero-Shot Text-to-Image Generation (2021, background): “We found that reducing the receptive field size for the convolutions around the relaxation led to it generalizing better to the true ELB.”
Neural Discrete Representation Learning (2017, methodology): “Typically, the posteriors and priors in VAEs are assumed normally distributed with diagonal covariance, which allows for the Gaussian reparametrisation trick to be used [32, 23].”
Unsupervised Visual Representation Learning by Context Prediction (2015, background): “To deal with these computational issues, a number of works, such as the wake-sleep algorithm [23], contrastive divergence [22], deep Boltzmann machines [45], and variational Bayesian methods [28, 43] use sampling to perform approximate inference.”
Learning to generate chairs with convolutional neural networks (2014, methodology): “This includes a wide variety of methods ranging from Gaussian mixture models [9, 31] to autoregressive models [19] and stochastic variations of neural networks [2, 11, 25, 17, 30].”