Learning Energy-Based Models in High-Dimensional Spaces with Multiscale Denoising-Score Matching

By Zengyi Li, Yubei Chen, Friedrich T. Sommer. 2019

34 citations recorded by the source corpus.

Energy-based models (EBMs) assign an unnormalized log probability to data samples. This functionality has a variety of applications, such as sample synthesis, data denoising, sample restoration, outlier detection, Bayesian reasoning and many more. But, the training of EBMs using standard maximum likelihood is extremely slow because it requires sampling from the model distribution. Score matching potentially alleviates this problem. In particular, denoising-score matching has been successfully used to train EBMs. Using noisy data samples with one fixed noise level, these models learn fast and yield good results in data denoising. However, demonstrations of such models in the high-quality sample synthesis of high-dimensional data were lacking. Recently, a paper showed that a generative model trained by denoising-score matching accomplishes excellent sample synthesis when trained with data samples corrupted with multiple levels of noise. Here we provide an analysis and empirical evidence showing that training with multiple noise levels is necessary when the data dimension is high. Leveraging this insight, we propose a novel EBM trained with multiscale denoising-score matching. Our model exhibits a data-generation performance comparable to state-of-the-art techniques such as GANs and sets a new baseline for EBMs. The proposed model also provides density information and performs well on an image-inpainting task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

Citations in the corpus

Score-based Generative Modeling in Latent Space (2021, methodology): “Our work builds on score-matching [29, 30, 31, 32, 33, 34, 35, 36, 37], speciﬁcally denoising score matching [22], which makes our work related to recent generative models using denoising score matching-and denoising diffusion-based objectives [3, 38, 1, 2, 6].”
Adversarial score matching and improved sampling for image generation (2020, background): “Li et al. (2019) and Song and Ermon (2020) found that the unconditional network produced better samples, while Ho et al. (2020) obtained better results than both of them using a conditional network.”
MCVD: Masked Conditional Video Diffusion for Prediction, Generation, and Interpolation (2022, background): “…be characterized as Denoising Diffusion Probabilistic Models (DDPMs) [Sohl-Dickstein et al., 2015, Ho et al., 2020, Dhariwal and Nichol, 2021], while another as Score-based Generative Models (SGMs) [Song and Ermon, 2019, Li et al., 2019, Song and Ermon, 2020, Jolicoeur-Martineau et al., 2021a].”
Variational (Gradient) Estimate of the Score Function in Energy-based Latent Variable Models (2020, unknown): “…minimizes the expected squared distance between the score function of the data distribution and that of the model distribution, and its variants (Kingma & LeCun, 2010; Vincent, 2011; Saremi et al., 2018; Song et al., 2019; Li et al., 2019; Pang et al., 2020) have shown promise in learning EBMs.”