Conditional Image Generation with PixelCNN Decoders

By Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Koray Kavukcuoglu, Oriol Vinyals, Alex Graves. 2016

2,801 citations recorded by the source corpus.

This work explores conditional image generation with a new image density model based on the PixelCNN architecture. The model can be conditioned on any vector, including descriptive labels or tags, or latent embeddings created by other networks. When conditioned on class labels from the ImageNet database, the model is able to generate diverse, realistic scenes representing distinct animals, objects, landscapes and structures. When conditioned on an embedding produced by a convolutional network given a single image of an unseen face, it generates a variety of new portraits of the same person with different facial expressions, poses and lighting conditions. We also show that conditional PixelCNN can serve as a powerful decoder in an image autoencoder. Additionally, the gated convolutional layers in the proposed model improve the log-likelihood of PixelCNN to match the state-of-the-art performance of PixelRNN on ImageNet, with greatly reduced computational cost.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Convolutional Sequence to Sequence Learning (2017, background): “A similar non-linearity has been introduced in Oord et al. (2016b) who apply tanh to A but Dauphin et al. (2016) shows that GLUs perform better in the context of language modelling.”
Denoising Diffusion Probabilistic Models (2020, neutral): “40 Gated PixelCNN [59] 4.”
Few-shot Autoregressive Density Estimation: Towards Learning to Learn Distributions (2017, methodology): “In this work, our focus is on extending autoregressive models to the few-shot setting, in particular PixelCNN (van den Oord et al., 2016).”
Axial Attention in Multidimensional Transformers (2019, methodology): “Multiscale PixelCNN (Reed et al., 2017) 3.95 3.70 PixelCNN/RNN (van den Oord et al., 2016a) 3.86 3.63 Gated PixelCNN (van den Oord et al., 2016b) 3.83 3.57 PixelSNAIL (Chen et al., 2018) 3.80 3.52 SPN (Menick & Kalchbrenner, 2018) 3.79 3.52 Image Transformer (Parmar et al., 2018) 3 (Ebert et al.,…”
Scaling Autoregressive Video Models (2019, methodology): “Our setup is closely related to that of Kalchbrenner et al. (2016), who extend work on pixel-level autoregressive image generation (van den Oord et al., 2016b;a) to videos.”
Image Transformer (2018, methodology): “Thus, parallelizable models that use convolutional neural networks such as the PixelCNN have recently received much more attention, and have now surpassed the PixelRNN in quality (van den Oord et al., 2016b).”
Jukebox: A Generative Model for Music (2020, methodology): “WaveNet (Oord et al., 2016) performs autoregressive sample-by-sample probabilistic modeling of raw waveform using a series of dilated convolutions to exponentially increase the context length.”
Representation Learning with Contrastive Predictive Coding (2018, methodology): “More recent advancements in autoregressive modeling such as masked convolutional architectures [18, 19] or self-attention networks [20] could help improve results further.”
Neural Discrete Representation Learning (2017, methodology): “The capacity of the PixelCNN we used was similar to those used by the authors of the PixelCNN paper [38].”
Improved Techniques for Training Score-Based Generative Models (2020, background): “Model Inception ↑ FID ↓ CIFAR-10 Unconditional PixelCNN [17] 4.”
Flow Contrastive Estimation of Energy-Based Models (2019, methodology): “This is also the case with auto-regressive models [35, 42, 38].”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, neutral): “PixelCNN [59] 4.”
BIVA: A Very Deep Hierarchy of Latent Variables for Generative Modeling (2019, neutral): “63 G ATED P IXEL CNN [56] = 3 .”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, methodology): “…begun to successfully model high dimensional raw observations from complex real-world datasets, from natural images and videos, to audio signals and natural language (Karras et al., 2017; Kalchbrenner et al., 2016b; van den Oord et al., 2016a; Kalchbrenner et al., 2016a; Vaswani et al., 2017).”
Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling (2018, methodology): “Analogous to the multi-scale ordering (van den Oord et al., 2016b), and depicted in 3(b), we can perform size upscaling explicitly , by training a single slice decoder on subimages and generate the ﬁrst slice of a subscale ordering from the single slice decoder itself.”
Glow: Generative Flow with Invertible 1x1 Convolutions (2018, methodology): “…modeling has experienced enormous leaps in capabilities in recent years, mostly with likelihood-based methods (Graves, 2013; Kingma and Welling, 2013, 2018; Dinh et al., 2014; van den Oord et al., 2016a) and generative adversarial networks (GANs) (Goodfellow et al., 2014) (see Section 4).”