Interpretation and Generalization of Score Matching

By Siwei Lyu. 2009

170 citations recorded by the source corpus.

Score matching is a recently developed parameter learning method that is particularly effective to complicated high dimensional density models with intractable partition functions. In this paper, we study two issues that have not been completely resolved for score matching. First, we provide a formal link between maximum likelihood and score matching. Our analysis shows that score matching finds model parameters that are more robust with noisy training data. Second, we develop a generalization of score matching. Based on this generalization, we further demonstrate an extension of score matching to models of discrete data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Score-based Generative Modeling in Latent Space (2021, methodology): “Our work builds on score-matching [29, 30, 31, 32, 33, 34, 35, 36, 37], specifically denoising score matching [22], which makes our work related to recent generative models using denoising score matching- and denoising diffusion-based objectives [3, 38, 1, 2, 6].”
A Kernelized Stein Discrepancy for Goodness-of-fit Tests (2016, methodology): “…divergence has been widely used for parameter estimation, ﬁnding the optimal q ( x ) that best ﬁts a sample { x i } by minimizing F ( p, q ) ; this yields the score matching methods developed in both parametric (Hyv¨arinen, 2005; Lyu, 2009) and non-parametric (Sriperumbudur et al., 2013) settings.”
Density Estimation in Infinite Dimensional Exponential Families (2013, methodology): “Under the conditions mentioned on p t and q t , it can be shown that See Theorem 1 in [26] for a proof.”
Offline Reinforcement Learning with Fisher Divergence Critic Regularization (2021, background): “…derivations establishing a connection between this gradient penalty and the Fisher divergence , which appears in the score matching and energy-based generative model literature (Lyu, 2012; Bao et al., 2020), interpreting the critic values as the energy function of a Boltzmann distribution.”
Maximum Likelihood Training of Score-Based Diffusion Models (2021, background): “[31] (Theorem 1) covers similar ground to the relative form of de Bruijn’s identity, but is perhaps the first to consider its implications for learning in probabilistic models by framing the discussion in terms of the score matching objective ([23], Eq.”
Deep Energy Estimator Networks (2018, background): “There is a formal link between maximum likelihood and score matching established by Lyu (2009).”
A Variational Perspective on Diffusion-Based Generative Models and Score Matching (2021, background): “To connect maximum likelihood and score matching, Durkan & Song (2021) shows that KL divergence can be represented as an integral of weighted Fisher divergence, generalizing the case of Lyu (2009) where the inference perturbation is a simple Brownian motion.”
Inductive Principles for Restricted Boltzmann Machine Learning (2010, methodology): “Recently, two new alternative principles have been proposed for dealing with the problem of intractable partition functions that are applicable to models of discrete data: ratio matching (RM) (Hyvärinen, 2007) and generalized score matching (GSM) (Lyu, 2009).”
A Connection Between Score Matching and Denoising Autoencoders (2011, methodology): “…more general frameworks of generalized score matching (Lyu, P. Vincent 2010; Marlin, Swersky, Chen, & de Freitas, 2010) and minimum probability ﬂow (Sohl-Dickstein et al., 2009), allowing generalizations of SM to discrete distributions (Hyv¨arinen, 2007b; Lyu, 2010; Sohl-Dickstein et al., 2009).”
On Maximum Likelihood Training of Score-Based Generative Models (2021, background): “Lyu [2009, Theorem 1] covers similar ground to the relative form of de Bruijn’s identity, but is perhaps the ﬁrst to consider its implications for learning in probabilistic models by framing the discussion in terms of the score matching objective [Hyv¨arinen, 2005, eq.”
Tackling the Generative Learning Trilemma with Denoising Diffusion GANs (2021, background): “Second, we are training the GAN on relatively smooth distributions, as the diffusion process is known as a smoothening process that brings the distributions of fake and real samples closer to each other (Lyu, 2012).”
Diffusion Models for Adversarial Purification (2022, background): “The proof follows (Song et al., 2021a; Lyu, 2009) that build connections between Fisher divergence and the “rate of change” in KL divergence by generalizing the de Bruijn’s identity (Barron, 1986), which we defer to Appendix A.1.”
Learning deep kernels for exponential family densities (2018, background): “Lyu (2009) connected it to maximum likelihood, arguing score matching ﬁnds a ﬁt which is robust to in-ﬁnitesimal amounts of noise.”
Gradient Estimators for Implicit Models (2017, methodology): “Generalisations of score matching methods are also presented in [24, 25].”
Score Matching Model for Unbounded Data Score (2021, methodology): “On the other hand, the score matching model minimizes the Fisher divergence [19], which is of order one (D(∇ log p,∇ log q)): the divergence contains the first derivative of two log-densities.”
How to Train Your Energy-Based Models (2021, background): “…of the connection between those two statistical divergences, as characterized by a relative version of the de Bruijin’s identity (Cover, 1999; Lyu, 2012): Here q t (˜ x ) and p θ ,t (˜ x ) denote smoothed versions of p data ( x ) and p θ ( x ) , resulting from adding Gaussian noise to x with…”
Unifying Non-Maximum Likelihood Learning Objectives with Minimum KL Contraction (2011, background): “Lemma 5 (Lyu [25]) For any two distributions p, q ∈ Ωd differentiable with regards to x, we have”
Score-Based Generative Modeling with Critically-Damped Langevin Diffusion (2021, background): “Generative modeling by learning stochastic processes has a long history (Movellan, 2008; Lyu, 2009; Sohl-Dickstein et al., 2011; Bengio et al., 2014; Alain et al., 2016; Goyal et al., 2017; Bordes et al., 2017; Song & Ermon, 2019; Ho et al., 2020).”