A Connection Between Score Matching and Denoising Autoencoders

By Pascal Vincent. 2011

2,271 citations recorded by the source corpus.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Deep AutoRegressive Networks (2013, background): “Recently several authors (Ranzato et al., 2007; Vincent et al., 2008; Vincent, 2011; Rifai et al., 2012; Bengio et al., 2013b) have developed probabilistic versions of regularised au-toencoders, along with means of generating samples from such models.”
Pylearn2: a machine learning research library (2013, methodology): “Other costs implemented in the library include dropout [27] , contrastive divergence [26], noise contrastive estimation [24], score matching [28], denoising s core matching [52], softmax log likelihood for classification with MLPs, and Gaussian log likelihood fo r regression with MLPs.”
Generalized Denoising Auto-Encoders as Generative Models (2013, background): “The last step in this development (Alain and Bengio, 2013) generalized the result from Vincent (2011) by showing that when a denoising auto-encoder (or a contractive auto-encoder with the contraction on the whole encode/decode reconstruction function) is trained with small Gaussian corruption and…”
Denoising Diffusion Probabilistic Models (2020, background): “In addition, we show that a certain parameterization of diffusion models reveals an equivalence with denoising score matching over multiple noise levels during training and with annealed Langevin dynamics during sampling (Section 3.2) [52, 58].”
Improved Techniques for Training Score-Based Generative Models (2020, methodology): “First, the score function is learned via denoising score matching [3, 4, 5].”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, background): “Denoising score matching Denoising score matching [61] is a variant of score matching that completely circumvents tr(∇xsθ(x)).”
A Theory of Generative ConvNet (2016, methodology): “The relationship between energy-based model with latent variables and auto-encoder was discovered by (Vincent, 2011) and (Swersky et al., 2011) via the score matching estimator (Hyvärinen, 2005).”
Classifier-Free Diffusion Guidance (2022, background): “This objective is denoising score matching [18] over multiple noise scales [16], and when p(λ) is uniform, the objective is proportional to the variational lower bound on the marginal log likelihood of the latent variable model ∫ pθ(x|z)pθ(z)dz, ignoring the term for the unspecified pθ(x|z) and for ”
Score-based Generative Modeling in Latent Space (2021, methodology): “[2] rely on denoising score matching [22] that converts the objective in Eq.”
Image Super-Resolution via Iterative Refinement (2021, methodology): “Score matching on noisy data, called denoising score matching [36], is equivalent to training a denoising autoencoder, and to DDPMs [1].”
Symbolic Music Generation with Diffusion Models (2021, background): “The following objective is simpler to train, resembles denoising score matching (Vincent, 2011; Song & Ermon, 2019) and was found to yield higher-quality samples:”
Score-Based Generative Modeling through Stochastic Differential Equations (2020, positive): “…Ermon (2019) propose to train a Noise Conditional Score Network (NCSN), denoted by s θ p x , σ q , with a weighted sum of denoising score matching (Vincent, 2011) objectives: Given sufﬁcient data and model capacity, the optimal score-based model s θ ˚ p x , σ q matches ∇ x log p σ p x q almost…”
Denoising Diffusion Implicit Models (2020, methodology): “The σ ( t ) -perturbed score function ∇ ¯ x log p t (¯ x ) is also a minimizer (from denoising score matching (Vincent, 2011)):”
VAEBM: A Symbiosis between Variational Autoencoders and Energy-based Models (2020, background): “These models are also based on denoising score matching (DSM) (Vincent, 2011), but do not parameterize any explicit energy function and instead directly model the vector-valued score function.”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “This reduction of generation to denoising is justified as optimizing a weighted variational lower bound on the data log likelihood under the diffusion model, or as a form of denoising score matching [72, 65, 28, 35].”
Elucidating the Design Space of Diffusion-Based Generative Models (2022, positive): “Thus, our work is in the context of denoising score matching [51].”
Representation Learning: A Review and New Perspectives (2012, methodology): “Indeed, as shown by one of us (Vincent, 2011) and others (Swersky et al., 2011), in the case of the restricted Boltzmann machine, training the model via an inductive principle known as score matching (Hyvärinen, 2005) (to be discussed in sec.”
BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Model (2019, background): “This brings the original BERT closer to a denoising autoencoder (Vincent et al., 2010), which could still be considered as training a Markov random ﬁeld however with (approximate) score matching (Vincent, 2011).”