Parallel Multiscale Autoregressive Density Estimation

By Scott E. Reed, Aäron van den Oord, Nal Kalchbrenner, Sergio Gomez Colmenarejo, Ziyun Wang, Yutian Chen, Dan Belov, Nando de Freitas. 2017

221 citations recorded by the source corpus.

PixelCNN achieves state-of-the-art results in density estimation for natural images. Although training is fast, inference is costly, requiring one network evaluation per pixel; O(N) for N pixels. This can be sped up by caching activations, but still involves generating each pixel sequentially. In this work, we propose a parallelized PixelCNN that allows more efficient inference by modeling certain pixel groups as conditionally independent. Our new PixelCNN model achieves competitive density estimation and orders of magnitude speedup - O(log N) sampling instead of O(N) - enabling the practical generation of 512x512 images. We evaluate the model on class-conditional image generation, text-to-image synthesis, and action-conditional video generation, showing that our model achieves the best results among non-pixel-autoregressive density models that allow efficient sampling.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generating Long Sequences with Sparse Transformers (2019, methodology): “…in modeling natural language (Jozefowicz et al., 2016) (Rad-ford et al., 2018) (Dai et al., 2018), raw audio (Van Den Oord et al., 2016) (Mehri et al., 2016), and images (Oord et al., 2016) (Menick & Kalchbrenner, 2018) (Salimans et al., 2017) (Reed et al., 2017) (Chen et al., 2017).”
Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions (2017, neutral): “This attention mechanism can also be straightforwardly applied to the multiscale PixelCNN architecture of Reed et al. (2017).”
Axial Attention in Multidimensional Transformers (2019, neutral): “Multiscale PixelCNN (Reed et al., 2017) 3.95 3.70 PixelCNN/RNN (van den Oord et al., 2016a) 3.86 3.63 Gated PixelCNN (van den Oord et al., 2016b) 3.83 3.57 PixelSNAIL (Chen et al., 2018) 3.80 3.52 SPN (Menick & Kalchbrenner, 2018) 3.79 3.52 Image Transformer (Parmar et al., 2018) 3 (Ebert et al.,…”
Scaling Autoregressive Video Models (2019, background): “An alternative approach is optionally hierarchical multi-scale generation, which has recently been explored for both image generation (Reed et al., 2017; De Fauw et al., 2019) as well as video generation (Mathieu et al., 2016).”
Predictive Sampling with Forecasting Autoregressive Models (2020, methodology): “Following van den Oord et al. (2017), we separate the training of autoencoder and ARM.”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, background): “Similar to the parallel multi-scale model of [29], SPN imposes a partitioning on the spatial dimensions, but unlike [29], SPN does not make the corresponding independence assumptions, whereby it trades sampling speed with density estimation performance and sample quality.”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, methodology): “Autoregressive Multiscale PixelCNN (Reed et al., 2017) – 3.”
Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling (2018, methodology): “Parallel Multiscale PixelCNN (Reed et al., 2017) is the only model in the literature which reports log-likelihood on 128× 128 ImageNet.”
Glow: Generative Flow with Invertible 1x1 Convolutions (2018, methodology): “2 (Reed et al., 2017) explores techniques for speeding up synthesis in AR models considerably; we leave the comparison to this line of work to future work.”
Efficient Neural Audio Synthesis (2018, background): “Multiple recent approaches have the goal of making sampling from sequential models more parallel (Reed et al., 2017; Gu et al., 2017; van den Oord et al., 2017).”
PixelSNAIL: An Improved Autoregressive Generative Model (2017, methodology): “Within the domain of modelling natural images, different extensions to neural autoregressive models were also explored: (Reed et al., 2017) explored mixing parallel generation into autoregressive ordering to speed up generation time, but still mostly rely on raster scan ordering; (Kolesnikov & Lampe”
Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Images (2020, methodology): “This can be viewed as an learned parallel multiscale generation method, unlike the handcrafted approaches of Kolesnikov & Lampert (2017); Menick & Kalchbrenner (2018); Reed et al. (2017).”
Cross-Modal Contrastive Learning for Text-to-Image Generation (2021, methodology): “Text-to-image synthesis Generating images from text descriptions has been quickly improved with deep generative models, including pixelCNN [55, 45], approximate Langevin sampling [34], variational autoencoders (VAEs) [21, 13] and Generative Adversarial Networks (GANs) [12, 44].”
AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, methodology): “[21] used conditional PixelCNN [27] to synthesize images from text with a multi-scale model structure.”
Generating Images with Sparse Representations (2021, methodology): “Sampling efficiency can be improved by substituting multi-head attention with multi-query attention (Shazeer, 2019), or by incorporating select conditional independence structure as in multiscale PixelCNN (Reed et al., 2017) or WaveRNN (Kalchbrenner et al.”
Object-Driven Text-To-Image Synthesis via Adversarial Training (2019, methodology): “There has been much research effort for this task through different approaches, such as variational inference [17, 6], approximate Langevin process [24], conditional PixelCNN via maximal likelihood estimation [26, 24], and conditional generative adversarial networks [23, 22, 32, 33].”
Inferring Semantic Layout for Hierarchical Text-to-Image Synthesis (2018, methodology): “[22] utilized sparse label maps like our method.”
Visual Foresight: Model-Based Deep Reinforcement Learning for Vision-Based Robotic Control (2018, neutral): “Several works have sought to use more complex distributions for future images, for example by using pixel autoregressive models [32], [36].”
Stochastic Variational Video Prediction (2017, methodology): “Reed et al. (2017) proposed a parallelized multi-scale algorithm that signiﬁcantly improves the training and prediction time but still requires more than a minute to generate one second of 64 × 64 video on a GPU.”
Stochastic Video Generation with a Learned Prior (2018, background): “Although impressive generative models of still images have been demonstrated (e.g. Reed et al. (2017); Karras et al. (2017)), these techniques do not extend to video sequences.”