Pixel Recurrent Neural Networks

By Aäron van den Oord, Nal Kalchbrenner, Koray Kavukcuoglu. 2016

2,869 citations recorded by the source corpus.

Modeling the distribution of natural images is a landmark problem in unsupervised learning. This task requires an image model that is at once expressive, tractable and scalable. We present a deep neural network that sequentially predicts the pixels in an image along the two spatial dimensions. Our method models the discrete probability of the raw pixel values and encodes the complete set of dependencies in the image. Architectural novelties include fast two-dimensional recurrent layers and an effective use of residual connections in deep recurrent networks. We achieve log-likelihood scores on natural images that are considerably better than the previous state of the art. Our main results also provide benchmarks on the diverse ImageNet dataset. Samples generated from the model appear crisp, varied and globally coherent.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Convolutional Sequence to Sequence Learning (2017, background): “A similar non-linearity has been introduced in Oord et al. (2016b) who apply tanh to A but Dauphin et al. (2016) shows that GLUs perform better in the context of language modelling.”
Neural Machine Translation in Linear Time (2016, methodology): “The scheme is repeated multiple times in the network always starting from a dilation rate of 1 (van den Oord et al., 2016a; Kalchbrenner et al., 2016b).”
XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, background): “With a deep root in density estimation 4 [4, 32, 24], language modeling has been a rapidly-developing research area [9, 1, 3].”
Denoising Diffusion Probabilistic Models (2020, background): “Generative adversarial networks (GANs), autoregressive models, flows, and variational autoencoders (VAEs) have synthesized striking image and audio samples [14, 27, 3, 58, 38, 25, 10, 32, 44, 57, 26, 33, 45], and there have been remarkable advances in energy-based modeling and score matching that ha”
Generating Long Sequences with Sparse Transformers (2019, methodology): “To address this, WaveNet (Van Den Oord et al., 2016) introduced dilated convolutions, which allowed the network to model long-range dependencies in a logarithmic number of layers.”
Neural Autoregressive Distribution Estimation (2016, methodology): “Even though ConvNADE+DeepNADE performs slightly worst than Pixel RNN, we note that our proposed approach is order-agnostic, whereas Pixel RNN requires a ﬁxed ordering.”
Evaluating Large Language Models Trained on Code (2021, background): “…(Mikolov et al., 2013; Sutskever et al., 2014; Dai & Le, 2015; Peters et al., 2018; Radford et al., 2018; Devlin et al., 2018), computer vision (Van Oord et al., 2016; Menick & Kalchbrenner, 2018; Chen et al., 2020; Bao et al., 2021), audio and speech processing (Oord et al., 2016; 2018; Dhari-wal…”
Axial Attention in Multidimensional Transformers (2019, methodology): “Multiscale PixelCNN (Reed et al., 2017) 3.95 3.70 PixelCNN/RNN (van den Oord et al., 2016a) 3.86 3.63 Gated PixelCNN (van den Oord et al., 2016b) 3.83 3.57 PixelSNAIL (Chen et al., 2018) 3.80 3.52 SPN (Menick & Kalchbrenner, 2018) 3.79 3.52 Image Transformer (Parmar et al., 2018) 3 (Ebert et al.,…”
Scaling Autoregressive Video Models (2019, background): “Our setup is closely related to that of Kalchbrenner et al. (2016), who extend work on pixel-level autoregressive image generation (van den Oord et al., 2016b;a) to videos.”
Image Transformer (2018, methodology): “Thus, parallelizable models that use convolutional neural networks such as the PixelCNN have recently received much more attention, and have now surpassed the PixelRNN in quality (van den Oord et al., 2016b).”
Contrastive Multiview Coding (2019, methodology): “Representations better suited to such tasks have been learnt using deep neural networks, starting with seminal techniques such as Boltzmann machines [54, 50], autoencoders [23], variational autoencoders [30], generative adversarial networks [20] and autoregressive models [45].”
Representation Learning with Contrastive Predictive Coding (2018, methodology): “Next, we use a PixelCNN-style autoregressive model [19] (a convolutional row-GRU PixelRNN [39] gave similar results) to make predictions about the latent activations in following rows top-to-bottom, visualized in Figure 4.”
Neural Discrete Representation Learning (2017, methodology): “PixelCNNs [29, 38] are convolutional autoregressive models which have also been used as distribution in the decoder of VAEs [15, 7].”
Predictive Sampling with Forecasting Autoregressive Models (2020, methodology): “Instead, we follow (van den Oord et al., 2016b) and use a one-hot encoding for inputs.”
Flow Contrastive Estimation of Energy-Based Models (2019, methodology): “This is also the case with auto-regressive models [35, 42, 38].”
Generative Modeling by Estimating Gradients of the Data Distribution (2019, methodology): “Recent progress is mainly driven by two approaches: likelihood-based methods [17, 29, 11, 60] and generative adversarial networks (GAN [15]).”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, methodology): “2a) to encode images onto a discrete latent space, and then we ﬁt a powerful PixelCNN prior over the discrete latent space induced by all the data.”