Deep Energy Estimator Networks

By Saeed Saremi, Arash Mehrjou, Bernhard Schölkopf, Aapo Hyvärinen. 2018

81 citations recorded by the source corpus.

Density estimation is a fundamental problem in statistical learning. This problem is especially challenging for complex high-dimensional data due to the curse of dimensionality. A promising solution to this problem is given here in an inference-free hierarchical framework that is built on score matching. We revisit the Bayesian interpretation of the score function and the Parzen score matching, and construct a multilayer perceptron with a scalable objective for learning the energy (i.e. the unnormalized log-density), which is then optimized with stochastic gradient descent. In addition, the resulting deep energy estimator network (DEEN) is designed as products of experts. We present the utility of DEEN in learning the energy, the score function, and in single-step denoising experiments for synthetic and high-dimensional data. We also diagnose stability problems in the direct estimation of the score function that had been observed for denoising autoencoders.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Flow Contrastive Estimation of Energy-Based Models (2019, background): “Another line of work is to estimate the parameters of EBM by score matching [26, 74, 63, 64].”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “We aim to train a conditional score network Similar to likelihood-based generative models and GANs, the design of model architectures plays an important role in generating high quality samples.”
Score-based Generative Modeling in Latent Space (2021, methodology): “Our work builds on score-matching [29, 30, 31, 32, 33, 34, 35, 36, 37], speciﬁcally denoising score matching [22], which makes our work related to recent generative models using denoising score matching-and denoising diffusion-based objectives [3, 38, 1, 2, 6].”
Image Super-Resolution via Iterative Refinement (2021, methodology): “Our approach is also linked to denoising score matching [35], [36], [37], [38] for training unnormalized energy”
Learning Energy-Based Models by Diffusion Recovery Likelihood (2020, neutral): “, 2020) and denoising score matching (Vincent, 2011; Song & Ermon, 2019; 2020; Saremi et al., 2018; Saremi & Hyvarinen, 2019).”
Sliced Score Matching: A Scalable Approach to Density and Score Estimation (2019, methodology): “Speciﬁcally, following the heuristic in Saremi et al. (2018) leads to σ = 1 .”
Solving Linear Inverse Problems Using the Prior Implicit in a Denoiser (2020, methodology): “More recently, [40, 19] directly train a neural network to estimate energy (a scalar value related to density) and equated its gradient to the score of the data.”
Learning the Stein Discrepancy for Training and Evaluating Energy-Based Models without Sampling (2020, methodology): “This changes the update rule to: This temperature-scaling approach has been used in most recent work on large-scale EBMs (Nijkamp et al., 2019a; Grathwohl et al., 2019; Nijkamp et al., 2019b; Du & Mor-datch, 2019).”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, background): “That is, su learns to estimate the noise thereby allowing it to be used as a generative model [190], [200].”
Efficient Learning of Generative Models via Finite-Difference Score Matching (2020, background): “However, when σ → 0, the variance of DSM could be large or even tend to infinity [66], requiring grid search or heuristics for choosing σ [53].”
Autoregressive Energy Machines (2019, methodology): “Deep energy estimator networks (Saremi et al., 2018) use an energy function implemented as a neural network, and train using the score-matching framework.”
Understanding Diffusion Models: A Unified Perspective (2022, methodology): “Fortunately, alternative techniques known as score matching [17, 18, 19, 20] have been derived to minimize this Fisher divergence without knowing the ground truth score, and can be optimized with stochastic gradient descent.”
Energy-Based Imitation Learning (2020, background): “4, DEEN finally converges in all experiments by differentiating the expert data.”
Annealed Denoising Score Matching: Learning Energy-Based Models in High-Dimensional Spaces (2019, methodology): “While learning the Parzen kernel density estimator under small noise was only proposed to analyze denoising autoencoder, Saremi et al. (2018) learned an energy-based model to match a Parzen estimator with a particular noise magnitude.”
Neural Empirical Bayes (2019, background): “Remark 2 (DEEN) For gaussian noise, the objective L = E X − (cid:98) x ( Y ) 2 is the same as the learning objective in “deep energy estimator networks” (DEEN) (Saremi et al., 2018) which itself was based on denoising score matching (Hyv¨arinen, 2005; Vincent, 2011).”
Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models (2020, background): “In particular, score matching (SM) (Hyvärinen, 2005), which minimizes the expected squared distance between the score function of the data distribution and that of the model distribution, and its variants (Kingma & LeCun, 2010; Vincent, 2011; Saremi et al., 2018; Song et al., 2019; Li et al., 2019; ”
How to Train Your Energy-Based Model for Regression (2020, methodology): “Recently, DSM has been successfully applied to generative image modeling [29, 46, 48].”
Stochastic Solutions for Linear Inverse Problems using the Prior Implicit in a Denoiser (2021, background): “[44, 26] trained a neural network to directly estimate energy (the negative log prior) and interpreted its gradient as the Score of the data.”
Improved Contrastive Divergence Training of Energy Based Models (2020, background): “One such approach is score matching, where the gradients of an energy function are trained to match the gradients of real data (Hyvärinen, 2005; Song & Ermon, 2019), or with a related denoising (Sohl-Dickstein et al., 2015; Saremi et al., 2018; Ho et al., 2020) approach.”