Deep Boltzmann Machines and the Centering Trick

By Grgoire Montavon, K. Müller. 2012

90 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Model Architectures, Probabilistic & Bayesian Methods, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Model Architectures

In Probabilistic & Bayesian Methods

In Signal Processing

Citations in the corpus

Neural Autoregressive Distribution Estimation (2016, background): “Work has also been directed towards proposing improved update rules or parameterization of the model’s energy function (Tieleman and Hinton, 2009; Cho et al., 2013; Montavon and M¨uller, 2012) as well as improved approximate inference of the hidden layers (Salakhutdinov and Larochelle, 2010).”
GSNs : Generative Stochastic Networks (2015, background): “…unsupervised learners are fairly well developed and used to pre-train these deep models, jointly training all the layers with respect to a single unsupervised criterion remains a challenge, with a few techniques arising to reduce that difficulty (Goodfellow et al., 2013; Montavon & Muller, 2012).”
Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) (2015, methodology): “Therefore tanh has been preferred over logistic functions (LeCun et al., 1991; 1998).”
Natural Neural Networks (2015, methodology): “This is analogous to the centering reparametrization for Deep Boltzmann Machines [13].”
Deep Learning of Representations: Looking Forward (2013, background): “io of largest to smallest eigenvalue). For example LeCun et al. (1998a) recommends centering and normalizing the inputs, an idea recently extended to hidden layers of Boltzmann machines with success (Montavon and Muller, 2012). A related idea that may have an impact on ill-conditioning is the idea o”
Deep learning for visual understanding: A review (2016, background): “For example, Montavon et al. [70] introduced the centering trick to improve the stability of a DBM and made it to be more discriminative and generative.”
Foundations and Advances in Deep Learning (2014, methodology): “βi = 〈xi〉dm was proposed in Publication VI, and Montavon and Müller (2012) proposed to use”
Metric-Free Natural Gradient for Joint-Training of Boltzmann Machines (2013, methodology): “All algorithms were run in conjunction with the centering strategy of Montavon and Muller [9], which proved crucial to successfully joint-train all layers of the DBM (even when using MFNG) 4.”
Deep Model Based Domain Adaptation for Fault Diagnosis (2017, unknown): “where α1 to α4 are learning rates that determined by LBFGS optimization algorithm adaptively [32].”
Deep Learning for Computer Vision: A Brief Review (2018, methodology): “These include accelerating inference by using separate models to initialize the values of the hidden units in all layers [47, 49], or other improvements at the pretraining stage [50, 51] or at the training stage [52, 53].”
Iterative Normalization: Beyond Standardization Towards Efficient Whitening (2019, methodology): “Some of these methods include activations centering in Restricted Boltzmann Machine [31]/feed-forward neural networks [46] and activations whitening [9, 29].”
On layer-wise representations in deep neural networks (2013, methodology): “Montavon and Müller (2012) propose the following reparameterization: α = 〈x〉 a = a+W (〈y〉 − β) β = 〈y〉 b = b+W⊤(〈x〉 − α) + V (〈z〉 − γ) γ = 〈z〉 c = c+ V ⊤(〈y〉 − β) A simplified training procedure for a three layers centered deep Boltzmann machine is shown in Algorithm 3. The algorithm is taken from t”
Deep Neural Networks Rival the Representation of Primate IT Cortex for Core Visual Object Recognition (2014, methodology): “The kernel analysis of deep Boltzmann machines in [68] also uses a mean squared loss function in the classification problem setting and is widely used in machine learning.”
Training restricted Boltzmann machines: An introduction (2014, methodology): “4), we arrive at AIS as proposed by Neal (2001). If we instead insert them into equation (9.”
Wasserstein Training of Restricted Boltzmann Machines (2016, methodology): “We use the centered parameterization of the RBM for gradient descent [13] [3].”