Sliced Score Matching: A Scalable Approach to Density and Score Estimation

By Yang Song, Sahaj Garg, Jiaxin Shi, Stefano Ermon. 2019

535 citations recorded by the source corpus.

Score matching is a popular method for estimating unnormalized statistical models. However, it has been so far limited to simple, shallow models or low-dimensional data, due to the difficulty of computing the Hessian of log-density functions. We show this difficulty can be mitigated by projecting the scores onto random vectors before comparing them. This objective, called sliced score matching, only involves Hessian-vector products, which can be easily implemented using reverse-mode automatic differentiation. Therefore, sliced score matching is amenable to more complex models and higher dimensional data compared to score matching. Theoretically, we prove the consistency and asymptotic normality of sliced score matching estimators. Moreover, we demonstrate that sliced score matching can be used to learn deep score estimators for implicit distributions. In our experiments, we show sliced score matching can learn deep energy-based models effectively, and can produce accurate score estimates for applications such as variational inference with implicit distributions and training Wasserstein Auto-Encoders.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Probabilistic & Bayesian Methods

Citations in the corpus

Improved Techniques for Training Score-Based Generative Models (2020, background): “In many situations the score function is easier to model and estimate than the original probability density function [3, 8].”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “As shown in [53], the term v∇xsθ(x)v can be efficiently computed by forward mode auto-differentiation.”
Score-Based Generative Modeling through Stochastic Differential Equations (2020, positive): “The score of a distribution can be estimated by training a score-based model on samples with score matching (Hyv¨arinen, 2005; Song et al., 2019a .”
WaveGrad: Estimating Gradients for Waveform Generation (2020, methodology): “, 2020) and score matching framework (Song et al., 2020; Song & Ermon, 2020).”
Maximum Likelihood Training of Score-Based Diffusion Models (2021, methodology): “With score matching techniques [56, 46], we can compute Eq.”
Learning to Efficiently Sample from Diffusion Probabilistic Models (2021, background): “…to score matching has been explored further by Song and Kingma [2021], where other score-matching techniques (e.g., sliced score matching, Song et al. et al., 2020a, Chen et al., 2021, Nichol and Dhariwal, 2021, San-Roman et al., 2021, Kong and Ping, 2021, Jolicoeur-Martineau et al.,…”
Learning Gradient Fields for Shape Generation (2020, neutral): “…loss λ ( σ i ) ; Point cloud encoder f φ ; A neural network s θ ; Total number of training iterations T ; Point cloud X t 1: for t ← 1 to T do for σ ∈ { σ i } ki =1 do 4: x X for i ∈ t do 5: ˜ x i ← x i + N (0 , σ 2 I ) for end 7: ( σ, 8: end for 9: } φ, θ ← Adam ( , φ, θ ) for t ← 0 to T − 1 do”
A Variational Perspective on Diffusion-Based Generative Models and Score Matching (2021, methodology): “A few alternative losses can be used, which are all equal to one another up to a constant, including implicit score matching (Hyvärinen & Dayan, 2005, ISM), sliced score matching (Song et al., 2020, SSM), and denoising score matching (Vincent, 2011, DSM).”
Learning Gradient Fields for Molecular Conformation Generation (2021, methodology): “Scorebased generative modeling (Song & Ermon, 2019; Song et al., 2020; Meng et al., 2020; Song & Ermon, 2020) is a class of generative models that approximate the score function of p(x) using neural networks and generate new samples with Langevin dynamics (Welling & Teh, 2011).”
Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models (2020, unknown): “In particular, score matching (SM) (Hyvärinen, 2005), which minimizes the expected squared distance between the score function of the data distribution and that of the model distribution, and its variants (Kingma & LeCun, 2010; Vincent, 2011; Saremi et al., 2018; Song et al., 2019; Li et al., 2019; ”
Efficient Learning of Generative Models via Finite-Difference Score Matching (2020, methodology): “[58] use random projections to avoid explicitly calculating the Hessian trace, so that the training objective only involves Hessian-vector products as follows:”
Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling (2020, methodology): “To make score matching efﬁcient, we also utilize Hutchin-son’s estimator when computing the Hessian trace, thus the objective we use to train is This modiﬁed score matching objective has been studied previously and is known as Sliced Score Matching (Song et al., 2019).”
Score-based diffusion models for accelerated MRI (2021, background): “Under some regularity conditions, sθ trained with denoising score matching will satisfy sθ∗ (x(t), t) = ∇x log pt(x) almost surely (Song et al., 2020).”
Understanding Diffusion Models: A Unified Perspective (2022, methodology): “Fortunately, alternative techniques known as score matching [17, 18, 19, 20] have been derived to minimize this Fisher divergence without knowing the ground truth score, and can be optimized with stochastic gradient descent.”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, methodology): “, score estimate) has established techniques such as score matching [101], denoising score matching [188, 189, 238], and sliced score matching [222], so we can directly employ one of them to train our noise-conditional score networks from perturbed data points.”
Annealed Denoising Score Matching: Learning Energy-Based Models in High-Dimensional Spaces (2019, background): “Previous efforts of learning energy-based models with score matching (Kingma and LeCun, 2010; Song et al., 2019) were either computationally intensive or unable to produce high-quality samples comparable to those obtained by other generative models such as GANs.”