Inductive Principles for Restricted Boltzmann Machine Learning

By Benjamin M. Marlin, Kevin Swersky, Nando de Freitas. 2010

167 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Neural Autoregressive Distribution Estimation (2016, background): “Other proposals include pseudo-likelihood (Besag, 1975; Marlin et al., 2010), score matching (Hyvärinen, 2005; Hyvärinen, 2007a,b), noise contrastive estimation (Gutmann and Hyvärinen, 2010) and probability flow minimization (Sohl-Dickstein et al.”
Representation Learning: A Review and New Perspectives (2012, positive): “Marlin et al. (2010) empirically compared all of these methods (except denoising score matching) on a range of classification, reconstruction and density modeling tasks and found that, in general, SML provided the best combination of overall performance and computational tractability.”
Discrete Variational Autoencoders (2016, methodology): “We train the resulting discrete VAEs on the MNIST [42], OMNIGLOT7 [43], and Caltech-101 Silhouettes datasets [44].”
VAE with a VampPrior (2017, background): “The results of our approach in comparison to the state-of-the-art methods are gathered in Table 2, 3, 4 and 5 for static and dynamic MNIST, OMNIGLOT and Caltech 101 Silhouettes, respectively.”
Iterative Neural Autoregressive Distribution Estimator NADE-k (2014, methodology): “We also evaluate the proposed NADE-k on Caltech-101 Silhouettes (Marlin et al., 2010), using the standard split of 4100 training samples, 2264 validation samples and 2307 test samples.”
DVAE++: Discrete Variational Autoencoders with Overlapping Transformations (2018, methodology): “In addition to statically bina-rized MNIST and OMNIGLOT, we test dynamically bi-narized MNIST (LeCun et al., 1998) and Caltech-101 sil-houettes (Marlin et al., 2010).”
PixelVAE++: Improved PixelVAE with Discrete Prior (2019, methodology): “The experiments are performed on statically and dynamically [50] binarized MNIST, Omniglot, and Caltech-101 silhouettes [51] and CIFAR-10 [52].”
Unsupervised Feature Learning and Deep Learning: A Review and New Perspectives (2012, unknown): “Marlin et al. (2010) provides an excellent survey of these methods and their relation to CD and SML.”
Enhanced Gradient and Adaptive Learning Rate for Training Restricted Boltzmann Machines (2011, methodology): “Remarkably, the classification accuracy improved by more than 5 % over the best result reported by Marlin et al. (2010).”
Enhanced Gradient for Training Restricted Boltzmann Machines (2013, methodology): “In contrast, Marlin et al. (2010) used an extensive validation to choose the right learning rate and other metaparameters and also applied an advanced optimization method as a fine-tuning method to the initial stochastic gradient descent optimization.”
Stochastic Ratio Matching of RBMs for Sparse High-Dimensional Inputs (2013, methodology): “This is much more expensive than SML as noted by Marlin et al. (2010).”
On Autoencoders and Score Matching for Energy Based Models (2011, background): “In such cases, stochastic algorithms can be applied to approximately maximize the likelihood and a variety of algorithms have been described and evaluated (Swersky et al., 2010; Marlin et al., 2010) in the literature including contrastive divergence (CD) (Hinton, 2002), persistent contrastive diverg”
On Tracking The Partition Function (2011, methodology): “5 Our models consisted of RBMs with 500 hidden units, trained using SML-APT [6] on the MNIST and Caltech Silhouettes [16] datasets.”
Deep Learning: A Comprehensive Overview on Techniques, Taxonomy, Applications and Research Directions (2021, background): “RBMs are a subset of Boltzmann machines that have a limit on the number of connections between the visible and hidden layers [77].”
Gradient Estimators for Implicit Models (2017, methodology): “Generalisations of score matching methods are also presented in [24, 25].”