On Autoencoders and Score Matching for Energy Based Models

By Kevin Swersky, Marc'Aurelio Ranzato, David Buchman, Benjamin M. Marlin, Nando de Freitas. 2011

98 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

Citations in the corpus

Generalized Denoising Auto-Encoders as Generative Models (2013, background): “This connection and its generalization to other energy functions, giving rise to the general denoising score matching training criterion, is discussed in several papers (Kingma and LeCun, 2010; Swersky et al. , 2011; Alain and Bengio, 2013).”
A Theory of Generative ConvNet (2016, methodology): “The relationship between energy-based model with latent variables and auto-encoder was discovered by (Vincent, 2011) and (Swersky et al., 2011) via the score matching estimator (Hyvärinen, 2005).”
Predicting Parameters in Deep Learning (2013, methodology): “Our approach is superficially similar to the factored RBM [21, 26], whose parameters form a 3tensor.”
Representation Learning: A Review and New Perspectives (2012, methodology): “Indeed, as shown by one of us (Vincent, 2011) and others (Swersky et al., 2011), in the case of the restricted Boltzmann machine, training the model via an inductive principle known as score matching (Hyvärinen, 2005) (to be discussed in sec.”
What regularized auto-encoders learn from the data-generating distribution (2012, background): “…score matching has been shown to be a consistent induction principle (Hyv¨arinen, 2005), it means that this denoising score matching (Vincent, 2011; Kingma and LeCun, 2010; Swersky et al., 2011) criterion recovers the underlying density, up to the smoothing induced by the noise of variance σ 2 .”
Learning Where to Attend with Deep Architectures for Image Tracking (2011, neutral): “For the ﬁrst layers, we use (factored)-RBMs (Hinton & Salakhutdinov, 2006; Ranzato & Hinton, 2010; Welling et al., 2005; Swersky et al., 2011), but autoencoders (Vincent et al., 2008), sparse coding (Olshausen & Field, 1996; Kavukcuoglu et al., 2009), two-layer ICA (K¨oster & Hyv¨arinen, 2007) and…”
Deep Learning of Representations: Looking Forward (2013, methodology): “Score matching (Hyvärinen, 2005) is an inductive principle that can be an interesting alternative to maximum likelihood, and several nconnections have been drawn between reconstruction error in auto-encoders and score matching (Swersky et al., 2011).”
Foundations and Advances in Deep Learning (2014, background): “Hence, this leads us to understand that learning the parameters of a GRBM by score matching is equivalent to learning the parameters of a single-layer autoencoder that has a hidden layer having nonlinear sigmoidal units by minimizing the reconstruction error with a regularization (Swersky et al., 20”
Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models (2020, unknown): “, a tractable posterior (Swersky et al., 2011) or a jointly exponential family model (Vértes et al.”
3D Shape Generation and Completion through Point-Voxel Diffusion (2021, unknown): “This form of supervision can be seen as supervision of the gradient of a log probability distribution [36] as in score matching EBM [15, 38].”
How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation (2014, methodology): “This is consistent with even earlier results showing that the denoising criterion is a regularized form of score matching (Vincent, 2011) called denoising score matching (Swersky et al. , 2011).”
Convex Two-Layer Modeling (2013, background): “A boolean representation can capture structures that range from a single latent clustering [11, 19, 20], by imposing the assumption that φ′1 = 1, to a general sparse code, by imposing the assumption that φ′1 = k for some small k [1, 4, 13].”
Estimating the Hessian by Back-propagating Curvature (2012, methodology): “Score matching is a simple alternative to maximum likelihood that has been widely used to train energy-based models (Köster and Hyvärinen, 2007; Swersky et al., 2011).”
Generalization Bounds for Neural Networks through Tensor Factorization (2015, methodology): “In addition, Swersky et al. (2011) analyze the score matching for latent energy-based models.”
Beating the Perils of Non-Convexity: Guaranteed Training of Neural Networks using Tensor Methods (2017, methodology): “The framework of score matching is popular for parameter estimation in probabilistic models (Hyvärinen, 2005; Swersky et al., 2011), where the criterion is to fit parameters based on matching the data score function.”
Learning unbiased features (2014, background): “Researchers have spent a great deal of effort developing new regularization schemes to improve the learned representation [11, 12, 9].”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, methodology): “An alternative approach is score matching [100] which is based on the idea of minimising the difference between the derivatives of the data and model’s log-density functions; the score function is defined as sðxÞ 1⁄4 rx ln pðxÞ which does not depend on the intractable denominator and can therefore b”
Score Function Features for Discriminative Learning: Matrix and Tensor Framework (2014, methodology): “ they also describe ways to estimate the second order score function. 18 Score matching: The framework of score matching is popular for parameter estimation in probabilistic models (Hyv¨arinen, 2005; Swersky et al., 2011), where the criterion is to ﬁt parameters is based on matching the data score f”