A Spectral Approach to Gradient Estimation for Implicit Distributions

By Jiaxin Shi, Jun Zhu. 2018

102 citations recorded by the source corpus.

Recently there have been increasing interests in learning and inference with implicit distributions (i.e., distributions without tractable densities). To this end, we develop a gradient estimator for implicit distributions based on Stein's identity and a spectral decomposition of kernel operators, where the eigenfunctions are approximated by the Nystrom method. Unlike the previous works that only provide estimates at the sample points, our approach directly estimates the gradient function, thus allows for a simple and principled out-of-sample extension. We provide theoretical results on the error bound of the estimator and discuss the bias-variance tradeoff in practice. The effectiveness of our method is demonstrated by applications to gradient-free Hamiltonian Monte Carlo and variational inference with implicit distributions. Finally, we discuss the intuition behind the estimator by drawing connections between the Nystrom method and kernel PCA, which indicates that the estimator can automatically adapt to the geometry of the underlying distribution.

Subfields: Artificial Intelligence, Environmental Engineering, Statistical and Nonlinear Physics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Environmental Engineering

In Statistical and Nonlinear Physics

Citations in the corpus

Sliced Score Matching: A Scalable Approach to Density and Score Estimation (2019, methodology): “We note that SSM outperforms Stein and Spectral in all cases.”
Functional Variational Bayesian Neural Networks (2019, methodology): “To approximate the marginal KL divergence gradients, we adopt the recently proposed spectral Stein gradient estimator (SSGE) [29].”
Understanding Variational Inference in Function-Space (2020, methodology): “4 Gaussian processes have been proposed for use in functional inference schemes both as priors (Sun et al., 2018) and as approximate posteriors (Ma et al., 2019).”
Spectral Inference Networks: Unifying Spectral Methods With Deep Learning (2018, methodology): “entiable layers in deep network architectures. Spectral decompositions have been used in combination with the kernelized Stein gradient estimator to better learn implicit generative models like GANs (Shi et al., 2018). While these use spectral methods to design or train neural networks, our work use”
Interacting Particle Solutions of Fokker–Planck Equations Through Gradient–Log–Density Estimation (2020, unknown): “We favoured this approach over other estimators [22,23] due to its flexibility to adapt to different function classes chosen to approximate the GLD.”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, methodology): “Various methods exist to estimate the score function including spectral approximation [194], sliced score matching [201], finite difference score matching [174], and notably”
Efficient Learning of Generative Models via Finite-Difference Score Matching (2020, background): “Different variants of SM have been proposed, including approximate back-propagation [25], curvature propagation [39], denoising score matching (DSM) [65], a bi-level formulation for latent variable models [3] and nonparametric estimators [35, 55, 59, 62, 74], but they may suffer from high computatio”
Spectral Inference Networks: Unifying Deep and Spectral Learning (2018, methodology): “Spectral decompositions have been used in combination with the kernelized Stein gradient estimator to better learn implicit generative models like GANs (Shi et al., 2018).”
Monte Carlo Gradient Estimation in Machine Learning (2019, methodology): “An alternative Stein-type gradient estimator using the spectral decomposition of kernels was developed by Shi et al. (2018), and Stein’s identity was also used in variational inference by Ranganath et al. (2016).”
Variational Implicit Processes (2018, methodology): “, 2018), where GP priors are directly used to regularize BNN training through gradient estimators (Shi et al., 2018).”