Connections Between Score Matching, Contrastive Divergence, and Pseudolikelihood for Continuous-Valued Variables

By Aapo Hyvärinen. 2007

69 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Neural Autoregressive Distribution Estimation (2016, background): “Other proposals include pseudo-likelihood (Besag, 1975; Marlin et al., 2010), score matching (Hyv¨arinen, 2005; Hyv¨arinen, 2007a,b), noise contrastive estimation (Gutmann and Hyv¨arinen, 2010) and probability ﬂow minimization (Sohl-Dickstein et al., 2011).”
A Theory of Generative ConvNet (2016, background): “Proposition 3 is related to score matching estimator (Hyvärinen, 2005), whose connection with contrastive divergence based on one-step Langevin was studied by (Hyvärinen, 2007).”
Minimum Probability Flow Learning (2009, background): “en by the data distribution from the similar score function given by the model distribution. It can be seen as an integration of the contrastive divergence gradient for inﬁnitesimal Langevin dynamics [8], as the limit of approximating the model distribution by patching together cutouts of the model ”
Deep Energy Estimator Networks (2018, methodology): “We also experimented with an alternative formulation of score matching by means of contrastive divergence (CD) with Langevin updates (Hyvärinen, 2007).”
A Connection Between Score Matching and Denoising Autoencoders (2011, background): “Previous studies have already pointed out connections between SM and contrastive divergence (Hyvarinen, 2007a ; Sohl-Dickstein et al, 2009 ), have connected SM to optimal denoising for gaussian noise with infinitesimal variance (Hyvarinen, 2008 ), and have shown that training gaussian binary RBM ”
Exponential Family Estimation via Adversarial Dynamics Embedding (2019, methodology): “The score matching (Hyvärinen, 2005) estimates the exponential family by minimizing the Fisher divergence, i.e. , As explained in Hyvärinen (2007), the objective (43) can be derived as the 2 nd-order Taylor approximation of the MLE with 1 -step Langevin Monte Carlo as the dual sampler.”
How to Train Your Energy-Based Models (2021, background): “…is given by After Taylor series expansion with respect to (cid:15) followed by some algebraic manipulations, the above equation can be transformed to the following (see Hyvarinen (2007)) When (cid:15) is suﬃciently small, it corresponds to the re-scaled gradient of the Score Matching objective.”
Unifying Non-Maximum Likelihood Learning Objectives with Minimum KL Contraction (2011, background): “For instance, in [15], it is shown that the parameter update in score matching [14] is equivalent to the parameter update in a version of contrastive divergence [12] that performs Langevin approximation instead of Gibbs sampling, and both are approximations to the parameter update of pseudo-likeliho”
Learning Deep Architectures for AI (2007, background): “Another alternative to Contrastive Divergence is Score Matching [84, 85, 86], a general approach to train energy-based models in which the energy can be computed tractably, but not the normalization constant Z.”