Improved Autoregressive Modeling with Distribution Smoothing

By Chenlin Meng, Jiaming Song, Yang Song, Shengjia Zhao, Stefano Ermon. 2021

24 citations recorded by the source corpus.

While autoregressive models excel at image compression, their sample quality is often lacking. Although not realistic, generated images often have high likelihood according to the model, resembling the case of adversarial examples. Inspired by a successful adversarial defense method, we incorporate randomized smoothing into autoregressive generative modeling. We first model a smoothed version of the data distribution, and then reverse the smoothing process to recover the original data distribution. This procedure drastically improves the sample quality of existing autoregressive models on several synthetic and real-world image datasets while obtaining competitive likelihoods on synthetic datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Statistical and Nonlinear Physics

In Generative Models

Citations in the corpus

On Fast Sampling of Diffusion Probabilistic Models (2021, background): “Diffusion models are connected with scored-based models (Song and Ermon, 2019, 2020; Song et al., 2020b), and there have been a series of research extending and improving diffusion models (Song et al., 2020b; Gao et al., 2020; Dhariwal and Nichol, 2021; San-Roman et al., 2021; Meng et al., 2021).”
Diffusion Normalizing Flow (2021, methodology): “The task is similar to image inpainting [27] but we use the whole masked forward trajectory instead of masked image [41, 17].”
Deep Generative Modelling: A Comparative Review of VAEs, GANs, Normalizing Flows, Energy-Based and Autoregressive Models (2021, methodology): “solution in this case is to add Gaussian noise, reducing the Lipschitz constant of the data distribution [152].”
Tackling the Generative Learning Trilemma with Denoising Diffusion GANs (2021, methodology): “Compared to Meng et al. (2021b), our model enjoys a 1100× speedup in generation, as it takes only 0.16s to generate one image at 256 resolution vs. 181s for Meng et al. (2021b).”
Diffusion Models: A Comprehensive Survey of Methods and Applications (2022, background): “(2021) [182] incorporates randomized smoothing into autoregressive generative modeling, in order to improve the sample quality.”
Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation (2023, unknown): “The type of data augmentation under consideration in this paper, namely additive Gaussian noise, is also a form of data distribution smoothing, which has been shown to improve sample quality in autoregressive models by Meng et al. [2021].”