Efficient Learning of Generative Models via Finite-Difference Score Matching

By Tianyu Pang, Kun Xu, Chongxuan Li, Yang Song, Stefano Ermon, Jun Zhu. 2020

65 citations recorded by the source corpus.

Several machine learning applications involve the optimization of higher-order derivatives (e.g., gradients of gradients) during training, which can be expensive in respect to memory and computation even with automatic differentiation. As a typical example in generative modeling, score matching (SM) involves the optimization of the trace of a Hessian. To improve computing efficiency, we rewrite the SM objective and its variants in terms of directional derivatives, and present a generic strategy to efficiently approximate any-order directional derivative with finite difference (FD). Our approximation only involves function evaluations, which can be executed in parallel, and no gradient computations. Thus, it reduces the total computational cost while also improving numerical stability. We provide two instantiations by reformulating variants of SM objectives into the FD forms. Empirically, we demonstrate that our methods produce results comparable to the gradient-based counterparts while being much more computationally efficient.

Subfields: Computational Mathematics, Computational Theory and Mathematics, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Theory and Mathematics

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

Score-based Generative Modeling in Latent Space (2021, methodology): “Our work builds on score-matching [29, 30, 31, 32, 33, 34, 35, 36, 37], specifically denoising score matching [22], which makes our work related to recent generative models using denoising score matching- and denoising diffusion-based objectives [3, 38, 1, 2, 6].”
Score-Based Generative Modeling through Stochastic Differential Equations (2020, neutral): “(7) uses denoising score matching, but other score matching objectives, such as sliced score matching (Song et al., 2019a) and ﬁnite-difference score matching (Pang et al., 2020) are also applicable here.”
Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models (2020, unknown): “…minimizes the expected squared distance between the score function of the data distribution and that of the model distribution, and its variants (Kingma & LeCun, 2010; Vincent, 2011; Saremi et al., 2018; Song et al., 2019; Li et al., 2019; Pang et al., 2020) have shown promise in learning EBMs.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, methodology): “Various methods exist to estimate the score function including spectral approximation [194], sliced score matching [201], finite difference score matching [174], and notably”