Efficient Gradient-Based Inference through Transformations between Bayes Nets and Neural Nets

By Diederik P. Kingma, Max Welling. 2014

64 citations recorded by the source corpus.

Hierarchical Bayesian networks and neural networks with stochastic hidden units are commonly perceived as two separate types of models. We show that either of these types of models can often be transformed into an instance of the other, by switching between centered and differentiable non-centered parameterizations of the latent variables. The choice of parameterization greatly influences the efficiency of gradient-based posterior inference; we show that they are often complementary to eachother, we clarify when each parameterization is preferred and show how inference can be made robust. In the noncentered form, a simple Monte Carlo estimator of the marginal likelihood can be used for learning the parameters. Theoretical results are supported by experiments.

Subfields: Artificial Intelligence, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Probabilistic & Bayesian Methods

Citations in the corpus

Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Model (2019, methodology): “For an observed image x , we reconstruct x by running gradient descent on the least squares loss function L ( z ) = x − M θ ( z ) 2 , initializing from z 0 ∼ p 0 ( z ) , and iterates z t + 1 = z t − η t L ′ ( z t ) . is we of k k For VAE, reconstruction readily available.”
Gradient Estimation Using Stochastic Computation Graphs (2015, background): “Kingma and Welling [11] consider a similar framework, although only with continuous latent variables, and point out that reparameterization can be used to to convert hierarchical Bayesian models into neural networks, which can then be trained by backpropagation.”
Reweighted Wake-Sleep (2014, background): “Past proposals to train Helmholtz machines and DBNs rely on maximizing a variational bound as proxy for the log-likelihood (Hinton et al. , 1995; Kingma and Welling, 2014b; Rezende et al. , 2014).”
Sylvester Normalizing Flows for Variational Inference (2018, background): “…the generative model and the inference network are given by neural networks, and sampling from the variational posterior is efficient through the noncentered parameterization (Kingma and Welling, 2014), also known as the reparameterization trick (Kingma and Welling, 2013; Rezende et al., 2014).”
Stick-Breaking Variational Autoencoders (2016, methodology): “We begin by reviewing the relevant background material on Variational Autoencoders (Kingma & Welling, 2014a), Stochastic Gradient Variational Bayes (also known as Stochastic Backpropagation ) (Kingma & Welling, 2014a; Rezende et al., 2014), and Stick-Breaking Processes (Ishwaran & James, 2001).”
Generalization without Systematicity: On the Compositional Skills of Sequence-to-Sequence Recurrent Networks (2017, methodology): “5 The ADAM optimization algorithm was used with default parameters, including a learning rate of 0.001 (Kingma & Welling, 2014).”
Still not systematic after all these years: On the compositional skills of sequence-to-sequence recurrent networks (2017, methodology): “The ADAM optimization algorithm was used with default parameters, including a learning rate of 0.001 (Kingma & Welling, 2014).”
Dirichlet Variational Autoencoder (2019, methodology): “A Variational Autoencoder (VAE) (Kingma & Welling, 2014c) brought success in deep generative models (DGMs) with a Gaussian distribution as a prior distribution (Jiang et al., 2017; Miao et al., 2016; 2017; Srivastava & Sutton, 2017).”
Monte Carlo Gradient Estimation in Machine Learning (2019, methodology): “…models in algorithms such as variational autoencoders, generative adversarial networks and generative stochastic networks (Rezende et al. (2014); Kingma and Welling (2014b); Goodfellow et al. (2014); Bengio et al. (2014)), in bandits and online learning (Hu et al., 2016), in econometrics and…”
Hyperspherical Variational Auto-Encoders (2018, methodology): “Applying the non-centered parameterization of the variational posterior (Kingma and Welling, 2014), further simplifies sampling and allows to reduce bias in calculating gradients for training.”
Early Stopping is Nonparametric Variational Inference (2015, background): “Kingma & Welling (2014) show how neural networks having unknown weights can be reformulated as neural networks having known weights but stochastic hidden units, and exploit this connection to preform efﬁcient gradient-based inference in Bayesian neural networks.”
How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation (2014, methodology): “The top layer may be allowed to have more noise and a different prior (e.g. as in Kingma and Welling (2014)). α =1 is reasonable choice, and controls the trade-off between reconstruction accuracy and the pressure to make Q ( h ) simple.”
Learning Energy-based Model with Flow-based Backbone by Neural Transport MCMC (2020, background): “Flow-based models have gained popularity in generative modeling [9, 10, 31, 18, 3, 35, 53] and variational inference [30, 49, 32, 28, 26].”
Modeling, Inference and Optimization With Composable Differentiable Procedures (2016, background): “Kingma & Welling [50] show how neural networks having unknown weights can be reformulated as neural networks having known weights”