PixelSNAIL: An Improved Autoregressive Generative Model

By Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, Pieter Abbeel. 2017

310 citations recorded by the source corpus.

Autoregressive generative models consistently achieve the best results in density estimation tasks involving high dimensional data, such as images or audio. They pose density estimation as a sequence modeling task, where a recurrent neural network (RNN) models the conditional distribution over the next element conditioned on all previous elements. In this paradigm, the bottleneck is the extent to which the RNN can model long-range dependencies, and the most successful approaches rely on causal convolutions, which offer better access to earlier parts of the sequence than conventional RNNs. Taking inspiration from recent work in meta reinforcement learning, where dealing with long-range dependencies is also essential, we introduce a new generative model architecture that combines causal convolutions with self attention. In this note, we describe the resulting model and present state-of-the-art log-likelihood results on CIFAR-10 (2.85 bits per dim) and $32 \times 32$ ImageNet (3.80 bits per dim). Our implementation is available at this https URL

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Generating Long Sequences with Sparse Transformers (2019, neutral): “…in modeling natural language (Jozefowicz et al., 2016) (Rad-ford et al., 2018) (Dai et al., 2018), raw audio (Van Den Oord et al., 2016) (Mehri et al., 2016), and images (Oord et al., 2016) (Menick & Kalchbrenner, 2018) (Salimans et al., 2017) (Reed et al., 2017) (Chen et al., 2017).”
Efficient Content-Based Sparse Attention with Routing Transformers (2020, neutral): “57 PixelSNAIL (Chen et al., 2018) - 3.”
Axial Attention in Multidimensional Transformers (2019, background): “…PixelCNN (Reed et al., 2017) 3.95 3.70 PixelCNN/RNN (van den Oord et al., 2016a) 3.86 3.63 Gated PixelCNN (van den Oord et al., 2016b) 3.83 3.57 PixelSNAIL (Chen et al., 2018) 3.80 3.52 SPN (Menick & Kalchbrenner, 2018) 3.79 3.52 Image Transformer (Parmar et al., 2018) 3 (Ebert et al., 2017).”
Scaling Autoregressive Video Models (2019, methodology): “To reduce the generally quadratic space complexity of the self-attention mechanism, we use block-local self-attention, generalizing the image generation approaches of Parmar et al. (2018) and Chen et al. (2018) to 3D volumes.”
Image Transformer (2018, neutral): “We trained the class-conditioned Image Transformer on CIFAR-10, achieving very similar log-likelihoods as in un-conditioned generation.”
Predictive Sampling with Forecasting Autoregressive Models (2020, background): “Neural network based ARMs currently hold state-of-the-art likelihood in image and audio domains (van den Oord et al., 2016a;b; Salimans et al., 2017; Chen et al., 2018; Menick & Kalchbrenner, 2018; Child et al., 2019).”
Generating Diverse High-Fidelity Images with VQ-VAE-2 (2019, methodology): “The prior over these discrete representations can be modeled with a state of the art PixelCNN [35, 36] with self-attention [38], called PixelSnail [6].”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, methodology): “Our best results were obtained by stacking convolutions and multi-head self attention into a gated residual network (Mishra et al., 2018; Chen et al., 2017), in a manner resembling the Transformer (Vaswani et al.”
Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling (2018, methodology): “…ImageNet 32x32 ImageNet 64x64 Gated PixelCNN (van den Oord et al., 2016c) 3.83 3.57 Parallel Multiscale (Reed et al., 2017) 3.95 3.70 PixelSNAIL (Chen et al., 2017) 3.80 - Image Transformer (Parmar et al., 2018) 3 ) and P ( x d 1 : d 2 h , w , c | x d 1 : d 2 < , x : d 1 ) respectively.…”
Autoregressive Quantile Networks for Generative Modeling (2018, methodology): “The recent PixelSNAIL model (Chen et al., 2017) achieves stateof-the-art modeling performance by enhancing PixelCNN with ELU nonlinearities, modified block structure, and an attention mechanism.”
Vector-quantized Image Modeling with Improved VQGAN (2021, methodology): “Then the second stage learns an autoregressive model, e.g., PixelSNAIL (Chen et al., 2018), to model the underlying distribution of token sequences.”
Score-based Generative Modeling in Latent Space (2021, background): “[81] XI Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel.”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, neutral): “, 2019) proposes using multi-scale hierarchical latents and SNAIL blocks (Chen et al., 2017) (and this setup has been applied to videos in (Walker et al.”
Taming Transformers for High-Resolution Image Synthesis (2020, methodology): “Convolutional architectures have been used for autoregressive modeling of images [61, 62, 10] but, for lowresolution images, previous works [48, 12, 24] demonstrated that transformers consistently outperform their convolutional counterparts.”
Improved Denoising Diffusion Probabilistic Models (2021, neutral): “(Ho et al., 2019) 3.69 3.08 PixelCNN (van den Oord et al., 2016c) 3.57 3.14 SPN (Menick & Kalchbrenner, 2018) 3.52 - NVAE (Vahdat & Kautz, 2020) - 2.91 Very Deep VAE (Child, 2020) 3.52 2.87 PixelSNAIL (Chen et al., 2018) 3.52 2.85 Image Transformer (Parmar et al., 2018) 3.48 2.90 Sparse Transformer ”
Self-Attention Generative Adversarial Networks (2018, background): “Recently, attention mechanisms have become an integral part of models that must capture global dependencies (Bahdanau et al., 2014; Xu et al., 2015; Yang et al., 2016; Gregor et al., 2015; Chen et al., 2018).”
Hierarchical Autoregressive Image Models with Auxiliary Decoders (2019, neutral): “Our autoregressive priors are very similar to those described in (Razavi et al., 2019), which are in turn closely related to PixelSNAIL (Chen et al., 2018).”