Neural Empirical Bayes

By Saeed Saremi, Aapo Hyvärinen. 2019

88 citations recorded by the source corpus.

We unify $\textit{kernel density estimation}$ and $\textit{empirical Bayes}$ and address a set of problems in unsupervised learning with a geometric interpretation of those methods, rooted in the $\textit{concentration of measure}$ phenomenon. Kernel density is viewed symbolically as $X\rightharpoonup Y$ where the random variable $X$ is smoothed to $Y= X+N(0,\sigma^2 I_d)$, and empirical Bayes is the machinery to denoise in a least-squares sense, which we express as $X \leftharpoondown Y$. A learning objective is derived by combining these two, symbolically captured by $X \rightleftharpoons Y$. Crucially, instead of using the original nonparametric estimators, we parametrize $\textit{the energy function}$ with a neural network denoted by $\phi$; at optimality, $\nabla \phi \approx -\nabla \log f$ where $f$ is the density of $Y$. The optimization problem is abstracted as interactions of high-dimensional spheres which emerge due to the concentration of isotropic gaussians. We introduce two algorithmic frameworks based on this machinery: (i) a "walk-jump" sampling scheme that combines Langevin MCMC (walks) and empirical Bayes (jumps), and (ii) a probabilistic framework for $\textit{associative memory}$, called NEBULA, defined a la Hopfield by the $\textit{gradient flow}$ of the learned energy to a set of attractors. We finish the paper by reporting the emergence of very rich "creative memories" as attractors of NEBULA for highly-overlapping spheres.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Probabilistic & Bayesian Methods

Citations in the corpus

Improved Techniques for Training Score-Based Generative Models (2020, background): “[9] noticed that adding an extra denoising step after the original annealed Langevin dynamics in [1], similar to [10, 11, 12], often significantly improves FID scores [13] without affecting the visual appearance of samples.”
Adversarial score matching and improved sampling for image generation (2020, background): “, taking the EDS) at the end of the Langevin sampling improves their quality (Saremi and Hyvarinen, 2019; Li et al., 2019; Kadkhodaie and Simoncelli, 2020).”
Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser (2020, background): “Recent results have derived relationships between Score matching density estimates and denoising [39, 40, 21, 41], and have used these relationships to make use of implicit prior information.”
Improved Autoregressive Modeling with Distribution Smoothing (2021, methodology): “We ﬁnd empirically that a pre-speciﬁed q (˜ x | x ) chosen by heuristics (Saremi & Hyvarinen, 2019; Garreau et al., 2017) is able to generate much better samples than training q (˜ x | x ) via ELBO.”
Stochastic Solutions for Linear Inverse Problems using the Prior Implicit in a Denoiser (2021, background): “Recent results have derived relationships between Score Matching density estimates and denoising, and have used these relationships to make use of implicit prior information [43, 44, 29, 45].”
Energy-Based Imitation Learning (2020, methodology): “This can be seen as a Parzen window estimation of p(x) with variance σ(2) as the smoothing parameter [38, 44].”
Learning Energy-Based Models in High-Dimensional Spaces with Multiscale Denoising-Score Matching (2019, methodology): “Our work builds on key ideas from Saremi et al. [26], Saremi and Hyvärinen [27] and Song and Ermon [28].”
Efficient Learning of Generative Models via Finite-Difference Score Matching (2020, methodology): “Our FD reformulation can also be used for score-based generative models [52, 57], where sθ(x) : R → R estimates ∇x log pdata(x) without modeling the likelihood by pθ(x).”
Annealed Denoising Score Matching: Learning Energy-Based Models in High-Dimensional Spaces (2019, methodology): “Saremi et al. (2018) and Saremi and Hyvarinen (2019) trained energy-based model with denoising score matching objective but the resulting models cannot perform sample synthesis from random noise initializations.”
PhysDiff: Physics-Guided Human Motion Diffusion Model (2022, methodology): “We then describe the details of the physics-based motion projection P π in Sec.”