PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modifications

By Tim Salimans, Andrej Karpathy, Xi Chen, Diederik P. Kingma. 2017

1,042 citations recorded by the source corpus.

PixelCNNs are a recently proposed class of powerful generative models with tractable likelihood. Here we discuss our implementation of PixelCNNs which we make available at this https URL Our implementation contains a number of modifications to the original model that both simplify its structure and improve its performance. 1) We use a discretized logistic mixture likelihood on the pixels, rather than a 256-way softmax, which we find to speed up training. 2) We condition on whole pixels, rather than R/G/B sub-pixels, simplifying the model structure. 3) We use downsampling to efficiently capture structure at multiple resolutions. 4) We introduce additional short-cut connections to further speed up optimization. 5) We regularize the model using dropout. Finally, we present state-of-the-art log likelihood results on CIFAR-10 to demonstrate the usefulness of these modifications.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Denoising Diffusion Probabilistic Models (2020, methodology): “To represent the reverse process, we use a U-Net backbone similar to an unmasked PixelCNN++ [52, 48] with group normalization throughout [66].”
High-Resolution Image Synthesis with Latent Diffusion Models (2021, background): “Because pixel based representations of images contain barely perceptible, high-frequency details [16,71], maximum-likelihood training spends a disproportionate amount of capacity on modeling them, resulting in long training times.”
Generating Long Sequences with Sparse Transformers (2019, neutral): “…in modeling natural language (Jozefowicz et al., 2016) (Radford et al., 2018) (Dai et al., 2018), raw audio (Van Den Oord et al., 2016) (Mehri et al., 2016), and images (Oord et al., 2016) (Menick & Kalchbrenner, 2018) (Salimans et al., 2017) (Reed et al., 2017) (Chen et al., 2017).”
Axial Attention in Multidimensional Transformers (2019, background): “Neural network models in this family have achieved state-of-the-art log likelihoods on highdimensional image and video datasets (van den Oord et al., 2016a; Chen et al., 2018; Menick & Kalchbrenner, 2018; Parmar et al., 2018; Child et al., 2019; Weissenborn et al., 2019; Salimans et al., 2017; Kalch”
Stand-Alone Self-Attention in Vision Models (2019, background): “A wide array of machine learning applications have leveraged convolutions to achieve competitive results including text-to-speech [36] and generative sequence models [37, 38].”
Zero-Shot Text-to-Image Generation (2021, background): “Likelihood objectives tend to prioritize modeling short-range dependencies between pixels (Salimans et al., 2017), so much of the modeling capacity would be spent capturing high-frequency details instead of the low-frequency structure that makes objects visually recognizable to us.”
Parallel WaveNet: Fast High-Fidelity Speech Synthesis (2017, methodology): “Since training a 65,536-way categorical distribution would be prohibitively costly, we instead modelled the samples with the discretized mixture of logistics distribution introduced in [23].”
Predictive Sampling with Forecasting Autoregressive Models (2020, methodology): “Autoregressive model architecture We base our PixelCNN implementation on a Pytorch implementation of PixelCNN++ (Salimans et al., 2017) by GitHub user pclucas14 (https://github.com/pclucas14/pixel-cnn-pp, commit 16c8b2f).”
Your Classifier is Secretly an Energy Based Model and You Should Treat it Like One (2019, background): “Nalisnick et al. (2018) showed that tractable deep generative models such as Kingma & Dhariwal (2018) and Salimans et al. (2017) can assign higher densities to OOD examples than in-distribution examples.”
Flow Contrastive Estimation of Energy-Based Models (2019, background): “Auto-regressive models [19, 56, 62], on the other hand, are (in principle) inefﬁcient to sample from, since synthesis requires computation that is proportional to the dimensionality of the data.”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, methodology): “In recent years autoregressive models, such as the PixelRNN and the PixelCNN [57, 45], have achieved superior likelihood performance and ﬂow-based models have proven efﬁcacy on large-scale natural image generation tasks [21].”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, methodology): “(Salimans et al., 2017) 2.92 – – Image Transformer (Parmar et al., 2018) 2.90 3.77 – PixelSNAIL (Chen et al., 2017) 2.85 3.80 3.52”
Autoregressive Quantile Networks for Generative Modeling (2018, methodology): “PixelCNN++ (Salimans et al., 2017) Finally, the implicit quantile formulation lifts a number of architectural restrictions of previous generative models.”
PixelSNAIL: An Improved Autoregressive Generative Model (2017, methodology): “These include: (i) PixelRNN [8], which uses LSTMs, (ii) PixelCNN [12] and PixelCNN++ [11], which only use causal convolutions, and (iii) Image Transformer [1], an attention-only architecture inspired by Vaswani et al. [14].”
Variational Diffusion Models (2021, methodology): “For comparison we also implemented Fourier features in a PixelCNN++ model [Salimans et al., 2017], where we do not see a benefit.”
Score-based Generative Modeling in Latent Space (2021, methodology): “[82] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma.”
Cascaded Diffusion Models for High Fidelity Image Generation (2021, positive): “Architectures The current best architectures for image diffusion models are U-Nets [19, 23], which are a natural choice to map corrupted data xt to reverse process parameters (µθ,Σθ) that have the same spatial dimensions as xt. Scalar conditioning, such as a class label or a diffusion timestep t, is”
High-Resolution Complex Scene Synthesis with Transformers (2021, background): “However, likelihood-based models have been demonstrated [25] to overspend capacity on short-range interactions of pixels when modeling images directly in pixel space.”