Neural Variational Inference and Learning in Belief Networks

By Andriy Mnih, Karol Gregor. 2014

734 citations recorded by the source corpus.

Highly expressive directed latent variable models, such as sigmoid belief networks, are difficult to train on large datasets because exact inference in them is intractable and none of the approximate inference methods that have been applied to them scale well. We propose a fast non-iterative approximate inference method that uses a feedforward network to implement efficient exact sampling from the variational posterior. The model and this inference network are trained jointly by maximizing a variational lower bound on the log-likelihood. Although the naive estimator of the inference network gradient is too high-variance to be useful, we make it practical by applying several straightforward model-independent variance reduction techniques. Applying our approach to training sigmoid belief networks and deep autoregressive networks, we show that it outperforms the wake-sleep algorithm on MNIST and achieves state-of-the-art results on the Reuters RCV1 document dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Generative Adversarial Nets (2014, neutral): “Other VAElike approaches exist [12, 22] but are less closely related to our method.”
Neural Discrete Representation Learning (2017, methodology): “…with continuous features have been the focus of many previous work [16, 39, 6, 9] however we concentrate on discrete representations [27, 33, 8, 28] which are potentially a more natural ﬁt for many of the modalities we are interested in. Language is inherently discrete, similarly…”
VQ-DRAW: A Sequential Discrete VAE (2020, background): “Other works do away with the reparameterization trick entirely, opting for other strategies for reducing variance in the variational lower bound [5] [6] [7].”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, background): “The three main classes of powerful explicit density models are autoregressive models [26, 57], flowbased models [8, 9, 21, 16], and probabilistic latent variable models [24, 40, 33].”
Learning Generative ConvNets via Multi-grid Modeling and Sampling (2017, methodology): “The model is usually trained together with an assisting model such as an inferential model as in the variational auto-encoder (VAE) [18, 38, 31], or a discriminative model as in the generative adversarial networks (GAN) [8, 6, 37].”
Density estimation using Real NVP (2016, background): “Recent advances in stochastic variational inference [27] and amortized inference [13, 43, 35, 49], allowed efficient approximate inference and learning of deep directed graphical models by maximizing a variational lower bound on the log-likelihood [45].”
A Theory of Generative ConvNet (2016, methodology): “One way to get around the intractable inference problem mentioned above is to use the wake-sleep algorithm [8] or the variational auto-encoder [14, 26, 22].”
Variational Inference with Normalizing Flows (2015, background): “For example, when compared to sigmoid belief networks that make use of mean-ﬁeld approximations, deep auto-regressive networks use a posterior approximation with an auto-regressive dependency structure that provides a clear improvement in performance (Mnih & Gregor, 2014).”
NICE: Non-linear Independent Components Estimation (2014, background): “However, recent advances in the framework of variational auto-encoders (VAE) - which come under a variety of names and variants (Kingma and Welling, 2014; Rezende et al., 2014; Mnih and Gregor, 2014; Gregor et al., 2014) - allowed effective approximate inference for training.”
DRAW: A Recurrent Neural Network For Image Generation (2015, methodology): “In this case the same reading and writing ﬁlters are used for all three channels.”
Generative Moment Matching Networks (2015, background): “…stochastic gradient descent, however they either require that the latent representation is continuous (Kingma & Welling, 2014), or require many secondary networks to sufﬁciently reduce the variance of gradient estimates in order to produce a sufﬁciently good learning signal (Mnih & Gregor, 2014).”
Foundation Posteriors for Approximate Probabilistic Inference (2022, neutral): “For discrete random variables, future work can study using the concrete relaxation [42] or the family of REINFORCE-based techniques [65, 46, 60].”
Categorical Reparameterization with Gumbel-Softmax (2016, methodology): “• NVIL (Mnih & Gregor, 2014) uses two baselines: (1) a moving average ¯ f of f to center the learning signal, and (2) an input-dependent baseline computed by a 1-layer neural network ﬁtted to f − ¯ f (a control variate for the centered learning signal itself).”
Self-Critical Sequence Training for Image Captioning (2016, background): “The recent discovery that REINFORCE with proper bias correction using learned “baselines” is effective has led to a flurry of work in applying REINFORCE to problems in RL, supervised learning, and variational inference [16, 17, 18].”
Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization (2018, background): “Note that more sophisticated baselines based on neural adaptation [22] or self-critic [23] can be also employed.”