A Mean Field Theory Learning Algorithm for Neural Networks

By Carsten Peterson, James R. Anderson. 1987

Subfields: Optimization & Training, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

In Probabilistic & Bayesian Methods

Citations in the corpus

Stochastic variational inference (2012, background): “In Michael Jordan’s lab, the seminal papers of Saul et al. (1996); Saul and Jordan (1996) and Jaakkola (1997) grew out of reading the statistical physics literature (Peterson and Anderson, 1987; Parisi, 1988).”
Variational Inference: A Review for Statisticians (2016, background): “Peterson and Anderson (1987) is arguably the first variational procedure for a particular model: a neural network.”
Uncertainty-guided Continual Learning with Bayesian Neural Networks (2019, background): “Alternatively, variational inference provides a faster solution to the same problem in which the posterior is approximated using optimization rather than being sampled from a chain [30, 11, 13, 12].”
Bayesian Compression for Deep Learning (2017, background): “It is well understood that variational inference can be reinterpreted from an MDL point of view [56, 72, 28, 30, 19].”
Practical Deep Learning with Bayesian Principles (2019, methodology): “References [1] James R Anderson and Carsten Peterson.”
Community detection in graphs (2009, methodology): “Lehmann andHansen [191] optimizedmodularity viamean field annealing [209], a deterministic alternative to simulated annealing [195].”
Iterative Neural Autoregressive Distribution Estimator NADE-k (2014, methodology): “(6)–(7): If we assume φ = σ and V = W>, the inference in the NADE-k is equivalent to performing k iterations of variational inference on an RBM for the missing values (Peterson and Anderson, 1987).”
Fast and Scalable Bayesian Deep Learning by Weight-Perturbation in Adam (2018, methodology): “The mean-field approximation has also been a popular tool from very early on (Saul et al., 1996; Anderson & Peterson, 1987).”
Noisy Natural Gradient as Variational Inference (2017, methodology): “Variational inference was ﬁrst applied to neural networks by Peterson (1987) and Hinton & Van Camp (1993).”
Learning Multilevel Distributed Representations for High-Dimensional Sequences (2007, methodology): “We use the mean-field equations [13] to compute pt fromp t−1 1 andV t 1 .”
A New Learning Algorithm for Mean Field Boltzmann Machines (2002, methodology): “An alternative to the slow Gibbs sampling is to approximate the averages using a fully factorized , mean field (MF) distribution (Peterson & Anderson, 1987).”
An Efficient Learning Procedure for Deep Boltzmann Machines (2012, methodology): “In an attempt to reduce the time required by the sampling process, Peterson and Anderson (1987) replaced Gibbs sampling with a simple mean-ﬁeld method that approximates a stationary distribution by replacing stochastic binary values with deterministic real-valued probabilities.”
Variational Cumulant Expansions for Intractable Distributions (2011, methodology): “The optimal parameters of the factorised model might therefore also be obtained by iterating, either synchronously or asynchronously (Peterson & Anderson, 1987), these (cid:12)xed point equations.”
Gradient calculations for dynamic recurrent neural networks: a survey (1995, methodology): “Two competing techniques for such problems are simulated annealing [150], [58] and mean eld theory [92].”