Mean-field theory of Boltzmann machine learning

By Toshiyuki TANAKA. 1998

I present a mean-field theory for Boltzmann machine learning, derived by employing Thouless-Anderson-Palmer free energy formalism to a full extent. Using the Plefka expansion an extended theory that takes higher-order correction to mean-field free energy formalism into consideration is presented, from which the mean-field approximation of general orders, along with the linear response correction, are derived by truncating the Plefka expansion up to desired orders. A theoretical foundation for an effective trick of using ``diagonal weights,'' introduced by Kappen and Rodr\'{\i}guez, is also given. Because of the finite system size and a lack of scaling assumptions on interaction coefficients, the truncated free energy formalism cannot provide an exact description in the case of Boltzmann machines. Accuracies of mean-field approximations of several orders are compared by computer simulations.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Deep Unsupervised Learning using Nonequilibrium Thermodynamics (2015, background): “A variety of analytic approximations exist which ameliorate, but do not remove, this tradeoff–for instance mean field theory and its expansions (T, 1982; Tanaka, 1998), variational Bayes (Jordan et al.”
Coevolutionary Landscape Inference and the Context-Dependence of Mutations in Beta-Lactamase TEM-1 (2015, neutral): “Following [76], also diagonal terms φii(a, b) = [C]i j(a, b) are included.”
Minimum Probability Flow Learning (2009, methodology): “Many approaches exist for approximate learning, including mean field theory and its expansions, variational Bayes techniques and a variety of sampling or numerical integration based methods (Tanaka, 1998; Kappen & Rodŕıguez, 1997; Jaakkola & Jordan, 1997; Haykin, 2008).”
Inverse statistical problems: from the inverse Ising problem to data science (2017, background): “…(58) In the inverse problem, the TAP free energy(57) gives an the equation for the couplings based on (42) Solving this quadratic equation gives the TAP reconstruction [103, 200] where we have chosen the solution that coincides with the mean-ﬁeld reconstruction when the magnetisations are zero.”
Linear Response Methods for Accurate Covariance Estimates from Mean Field Variational Bayes (2015, background): “[16] states that linear response methods could be applied to general exponential family models but works out details only for Boltzmann machines.”
The Natural Gradient by Analogy to Signal Whitening, and Recipes and Tricks for its Use (2012, background): “…ﬂow [Sohl-Dickstein et al., 2011b, Sohl-Dickstein et al., 2011a], contrastive divergence [Welling and Hinton, 2002, Hinton, 2002], score matching [Hyv¨arinen, 2005], mean ﬁeld theory, and variational bayes [Tanaka, 1998, Kappen and Rodriguez, 1997, Jaakkola and Jordan, 1997, Haykin, 2008]).”
PUMA: A Programmable Ultra-efficient Memristor-based Accelerator for Machine Learning Inference (2019, background): “tiveAdversarialNetworks(GANs) are composed of two neural networks (MLP, LSTM, CNN, etc.) which compete to reach equilibrium [44]. Restricted Boltzmann Machines (RBM) [102] and Boltzmann Machines (BM) [104] are commonly used in unsupervised learning tasks for energyminimization. While RBM involves we”
Predicting protein–ligand affinity with a random matrix framework (2016, background): “However, the classification accuracy does not significantly improve if the energy is estimated using the leading-order mean-field approximation (37).”