Mean Field Theory for Sigmoid Belief Networks

By Lawrence K. Saul, Tommi S. Jaakkola, Michael I. Jordan. 1996

We develop a mean field theory for sigmoid belief networks based on ideas from statistical mechanics. Our mean field theory provides a tractable approximation to the true probability distribution in these networks; it also yields a lower bound on the likelihood of evidence. We demonstrate the utility of this framework on a benchmark problem in statistical pattern recognition-the classification of handwritten digits.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Density estimation using Real NVP (2016, background): “Its success in leveraging recent advances in backpropagation [51, 39] in deep neural networks resulted in its adoption for several applications ranging from speech synthesis [12] to language modeling [8].”
Stochastic variational inference (2012, background): “In Michael Jordan’s lab, the seminal papers of Saul et al. (1996); Saul and Jordan (1996) and Jaakkola (1997) grew out of reading the statistical physics literature (Peterson and Anderson, 1987; Parisi, 1988).”
One-Shot Generalization in Deep Generative Models (2016, background): “Models that are directed graphical models have recently become popular in machine learning and include discrete latent variable models such as sigmoid belief networks and deep auto-regressive networks (Saul et al., 1996; Gregor et al., 2014), or continuous latent variable models such as non-linear G”
Variational Inference: A Review for Statisticians (2016, background): “This article, along with insights from statistical mechanics (Parisi 1988), led to a flurry of variational inference procedures for a wide class of models (Saul, Jaakkola, and Jordan 1996; Jaakkola and Jordan 1996, 1997; Ghahramani and Jordan 1997; Jordan et al. 1999).”
MuProp: Unbiased Backpropagation for Stochastic Neural Networks (2015, methodology): “Algorithms based on Markov chain Monte Carlo (Neal, 1992) and mean-field inference (Saul et al., 1996) are theoretically well-grounded, but do not scale well to large models.”
Disentangling Factors of Variation via Generative Entangling (2012, methodology): “Following the variational EM training approach (Saul et al., 1996), we alternately maximize the lower bound L ( Q v ) with respect to the variational parameters ˆ f , ˆ g and ˆ h (E-step) and maximizing L ( Q v ) with respect to the model parameters (M-step).”
Weight Uncertainty in Neural Networks (2015, background): “…θ q ( w | θ ) log q ( w | θ ) P ( w ) P ( D| w ) d w The resulting cost function is variously known as the variational free energy (Neal and Hinton, 1998; Yedidia et al., 2000; Friston et al., 2007) or the expected lower bound (Saul et al., 1996; Neal and Hinton, 1998; Jaakkola and Jordan, 2000).”
Stochastic Back-propagation and Variational Inference in Deep Latent Gaussian Models (2014, background): “Copyright 2014 by the author(s). ilar latent variable models (Dayan et al., 1995; Frey, 1996; Saul et al., 1996; Bartholomew & Knott, 1999; Uria et al., 2013; Gregor et al., 2013) can be easily sampled from, but in most cases, eﬃcient inference algorithms have remained elusive.”
Boundary-Seeking Generative Adversarial Networks (2017, neutral): “This situation is analogous to that in training Helmholtz machines with discrete hidden variables (i.e., sigmoid belief networks (SBNs, Saul et al., 1996)), where advanced learning algorithms are necessary to train an approximate inference network (Mnih & Gregor, 2014; Bornschein & Bengio, 2014).”
Techniques for Learning Binary Stochastic Feedforward Neural Networks (2014, methodology): “The early work on SFNNs approached the inference of h using Gibbs sampling (Neal, 1990, 1992) or mean ﬁeld (Saul et al. , 1996), which both have their downsides.”
Reweighted Wake-Sleep (2014, methodology): “In the following we will describe experiments containing three kinds of layers: Signoid Belief Network (SBN): A SBN layer (Saul et al., 1996) is a directed graphical model with independent variables xi given the parents y.”
Practical Deep Learning with Bayesian Principles (2019, methodology): “, by using MCMC methods [41], Laplace’s method [35], and variational inference (VI) [18, 2, 49, 1].”
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, methodology): “The mean-field approximation has also been a popular tool from very early on (Saul et al., 1996; Anderson & Peterson, 1987).”
Feature Learning in Deep Neural Networks - A Study on Speech Recognition Tasks (2013, methodology): “In the ﬁnal layer, the class posterior probabilities are computed as a multinomial distribution Note that the equality between p y | x ( y = s | x ) and p L ( y = s | v L ) is valid by making a mean-ﬁeld approximation [14] at each hidden layer.”
Dynamical Isometry and a Mean Field Theory of RNNs: Gating Enables Signal Propagation in Recurrent Neural Networks (2018, background): “Mean ﬁeld theory also has long been used to study Boltzmann machines (Ackley et al.) and sigmoid belief networks (Saul et al., 1996).”
Deep Exponential Families (2014, methodology): “Here, instead, we develop variational methods [35] that are applicable to general DEFs.”
Importance Weighting and Variational Inference (2018, background): “When the integral p✓(x) = R p✓(z,x)dz is intractable, one can optimize the lower-bound Eq(z) log (p✓(z,x)/q(z)) instead [22], over both ✓ and the parameters of q.”
Max-Margin Deep Generative Models (2015, background): “Though the latent representations can be used with a downstream classifier to make predictions, it is often beneficial to learn a joint model that considers both the input and response variables.”