Regularized estimation of image statistics by Score Matching

By Diederik P. Kingma, Yann LeCun. 2010

89 citations recorded by the source corpus.

Subfields: Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Probabilistic & Bayesian Methods

Citations in the corpus

Generalized Denoising Auto-Encoders as Generative Models (2013, background): “This connection and its generalization to other energy functions, giving rise to the general denoising score matching training criterion, is discussed in several papers (Kingma and LeCun, 2010; Swersky et al. , 2011; Alain and Bengio, 2013).”
Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, methodology): “This is similar to the objective of Score Matching (Hyvärinen, 2005) which can also be used to train EBMs (Kingma & Lecun, 2010; Song & Ermon, 2019).”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “We aim to train a conditional score network Similar to likelihood-based generative models and GANs, the design of model architectures plays an important role in generating high quality samples.”
Score-based Generative Modeling in Latent Space (2021, methodology): “Our work builds on score-matching [29, 30, 31, 32, 33, 34, 35, 36, 37], specifically denoising score matching [22], which makes our work related to recent generative models using denoising score matching- and denoising diffusion-based objectives [3, 38, 1, 2, 6].”
Representation Learning: A Review and New Perspectives (2012, methodology): “One potential drawback of these methods is that depending on the parametrization of the energy function, their computational requirements may scale up to O(nd) worse than CD, SML, FPCD, or denoising score matching (Kingma and LeCun, 2010; Vincent, 2011), discussed below.”
What regularized auto-encoders learn from the data-generating distribution (2012, background): “…score matching has been shown to be a consistent induction principle (Hyv¨arinen, 2005), it means that this denoising score matching (Vincent, 2011; Kingma and LeCun, 2010; Swersky et al., 2011) criterion recovers the underlying density, up to the smoothing induced by the noise of variance σ 2 .”
Deep Learning of Representations: Looking Forward (2013, background): “This connection and its generalization to other energy functions, giving rise to the general denoising score matching training criterion, is discussed in several papers (Kingma and LeCun, 2010; Swersky et al., 2011; Alain and Bengio, 2013).”
Sliced Score Matching: A Scalable Approach to Density and Score Estimation (2019, methodology): “Several approaches have been proposed to alleviate this difﬁculty: Kingma & LeCun (2010) propose approximate backpropagation for computing the trace of the Hessian; Martens et al. (2012) develop curvature propagation, a fast stochastic estimator for the trace in score matching; and Vincent (2011)…”
Deep Energy Estimator Networks (2018, background): “8 written in terms of the energy ψ ( x ; θ ) = −∇ x E ( x ; θ ) and expanded in the vector coordinates i reduces to (Kingma & LeCun, 2010) considered regularizing this objective where they arrived at regularized form of the score matching objective given by: The second term above was obtained by a…”
How to Train Your Energy-Based Models (2021, background): “Kingma and LeCun (2010) showed that the objective with noisy data can be approximated by the noiseless Score Matching objective of Eq.”
Learning deep kernels for exponential family densities (2018, methodology): “Given k , z , and λ , Proposition 3 (Appendix B) shows we can ﬁnd the optimal α by solving an M × M linear system in O ( M 2 ND + M 3 ) time: the Previous attempts at deep score matching Kingma and LeCun (2010) used score matching to train a network to output an energy.”
On Autoencoders and Score Matching for Energy Based Models (2011, methodology): “Autoencoder neural networks are another class of models that are often used to model high-dimensional realvalued data (Hinton & Zemel, 1994; Vincent et al., 2008; Vincent, 2011; Kingma & LeCun, 2010).”
A Connection Between Score Matching and Denoising Autoencoders (2011, background): “Also, since Kingma and LeCun (2010) showed that it may be preferable to use a regularized criterion (that they derived from smoothed empirical distribution q σ ), it is likely that our q σ -basedcriteriamay,for σ > 0,yieldbettergeneralizationperformance than the J ISMq 0 advocated in Hyv¨arinen…”
Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models (2020, unknown): “…minimizes the expected squared distance between the score function of the data distribution and that of the model distribution, and its variants (Kingma & LeCun, 2010; Vincent, 2011; Saremi et al., 2018; Song et al., 2019; Li et al., 2019; Pang et al., 2020) have shown promise in learning EBMs.”
Exponential Family Estimation via Adversarial Dynamics Embedding (2019, background): “Unfortunately, it requires third order derivatives for optimization, which becomes prohibitive for large models (Kingma and LeCun, 2010; Li et al., 2019).”
Regularized Auto-Encoders Estimate Local Statistics (2012, background): “Since score matching has been shown to be a consistent induction principle (Hyvärinen, 2005), it means that this denoising score matching (Vincent, 2011; Kingma and LeCun, 2010; Swersky et al., 2011) criterion recovers the underlying density, up to the smoothing induced by the noise of variance σ(2)”
Learning Energy-Based Models in High-Dimensional Spaces with Multiscale Denoising-Score Matching (2019, background): “Second, a random vector is always close to orthogonal to a ﬁxed vector [33].”
Learning Deep Energy Models (2011, methodology): “A potential alternative to samplingbased learning is to use score matching to train our models; for example, Kingma & LeCun (2010) showed how score matching can be applied to arbitary energy functions while Köster & Hyvärinen (2010) showed how to train both layers of a two layer ICA-based model.”
Deep Structured Energy Based Models for Anomaly Detection (2016, background): “Methodology-wise, there is also a recent surge of training EBMs with score matching (Vincent, 2011; Swer-sky et al., 2011; Kingma & Cun, 2010).”
On the Expressive Power of Deep Architectures (2011, background): “…generative model of the distribution of the uncorrupted examples (Vincent et al., 2008; Vincent, 2011), and in particular through the Score Matching proxy for log-likelihood (Vincent, 2011): the denoising error corresponds to a form of regularized score matching criterion (Kingma and LeCun, 2010).”