Metric-Free Natural Gradient for Joint-Training of Boltzmann Machines

By Guillaume Desjardins, Razvan Pascanu, Aaron C. Courville, Yoshua Bengio. 2013

32 citations recorded by the source corpus.

This paper introduces the Metric-Free Natural Gradient (MFNG) algorithm for training Boltzmann Machines. Similar in spirit to the Hessian-Free method of Martens [8], our algorithm belongs to the family of truncated Newton methods and exploits an efficient matrix-vector product to avoid explicitely storing the natural gradient metric $L$. This metric is shown to be the expected second derivative of the log-partition function (under the model distribution), or equivalently, the variance of the vector of partial derivatives of the energy function. We evaluate our method on the task of joint-training a 3-layer Deep Boltzmann Machine and show that MFNG does indeed have faster per-epoch convergence compared to Stochastic Maximum Likelihood with centering, though wall-clock performance is currently not competitive.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Revisiting Natural Gradient for Deep Networks (2013, background): “For convenience we show the formulas for two typical pairs of output activations and corresponding error function, where J y stands for the Jacobian matrix ∂ y ∂θ , y being the output layer.”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “FActorized Natural Gradient (FANG) (Grosse & Salakhudinov, 2015) estimates the natural gradient via an approximation of the Fisher information matrix by a Gaussian graphical model.”
Kronecker-factored Curvature Approximations for Recurrent Neural Networks (2018, methodology): “…matrix approximations such as TONGA (Le Roux et al., 2008) and the Hessian-free (HF) approach (Martens, 2010; Martens & Sutskever, 2011; 2012; Desjardins et al., 2013; Sainath et al., 2013) demonstrated the potential of such methods, they never achieved wide adoption due to issues of…”
Training Deep and Recurrent Networks with Hessian-Free Optimization (2012, methodology): “HF has also served as a basis or inspiration for several new optimization approaches (Chapelle and Erhan, 2011; Vinyals and Povey, 2012; Kiros, 2013; Sainath et al., 2013a; Bengio et al., 2013; Desjardins et al., 2013), and in particular the recent method of Martens and Grosse (2015) known as K-FAC.”
Training Restricted Boltzmann Machines (2015, methodology): “For binary RBMs the entries of the Fisher information matrix (Amari et al., 1992; Desjardins et al., 2013; Ollivier et al., 2013) are given by”
Scaling up Natural Gradient by Sparsely Factorizing the Inverse Fisher Matrix (2015, methodology): “Montavon & Muller (2012) used the centering trick to train deep Boltzmann machines without a pre-training step, and Desjardins et al. (2013) found that it was critical to achieving good performance with metric-free natural gradient.”