Image Transformer

By Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, Dustin Tran. 2018

1,925 citations recorded by the source corpus.

Image generation has been successfully cast as an autoregressive sequence generation or transformation problem. Recent work has shown that self-attention is an effective way of modeling textual sequences. In this work, we generalize a recently proposed model architecture based on self-attention, the Transformer, to a sequence modeling formulation of image generation with a tractable likelihood. By restricting the self-attention mechanism to attend to local neighborhoods we significantly increase the size of images the model can process in practice, despite maintaining significantly larger receptive fields per layer than typical convolutional neural networks. We propose another extension of self-attention allowing it to efficiently take advantage of the two-dimensional nature of images. While conceptually simple, our generative models trained on two image data sets are competitive with or significantly outperform the current state of the art in autoregressive image generation on two different data sets, CIFAR-10 and ImageNet. We also present results on image super-resolution with a large magnification ratio, applying an encoder-decoder configuration of our architecture. In a human evaluation study, we show that our super-resolution models improve significantly over previously published autoregressive super-resolution models. Images they generate fool human observers three times more often than the previous state of the art.

Subfields: Artificial Intelligence, Biophysics, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biophysics

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “Parmar et al. (2018) applied the self-attention only in local neighborhoods for each query pixel instead of globally.”
End-to-End Object Detection with Transformers (2020, methodology): “Transformers are now replacing RNNs in many problems in natural language processing, speech processing and computer vision [8,27,45,34,31].”
Generating Long Sequences with Sparse Transformers (2019, methodology): “(Parmar et al., 2018) uses blocks of local attention to apply Transformers to images.”
Music Transformer: Generating Music with Long-Term Structure (2018, methodology): “With a larger dataset, local attention may fare well as it allows for much deeper models and longer sequences, as seen in text and image generation work (Liu et al., 2018; Parmar et al., 2018)).”
Rethinking Attention with Performers (2020, methodology): “Most approaches restrict the attention mechanism to attend to local neighborhoods (Parmar et al., 2018) or incorporate structural priors on attention such as sparsity (Child et al.”
Reformer: The Efficient Transformer (2020, methodology): “Up to 11 thousand tokens of text in a single example were processed in (Liu et al., 2018) and when processing other modalities, like music (Huang et al., 2018) and images (Parmar et al., 2018), even longer sequences are commonplace.”
Root Mean Square Layer Normalization (2019, methodology): “Due to its simplicity and requiring no dependencies among training cases, LayerNorm has been widely applied to different neural architectures, which enables remarkable success on various tasks ranging from computer vision (Parmar et al., 2018; Sharma et al., 2018), speech recognition (Zhou et al.”
Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, background): “[18, 13, 21, 17, 15] on the other hand restrict receptive ﬁeld of self-attention and use it in a convolutional manner.”
Exploring Self-Attention for Image Recognition (2020, neutral): “This has inspired applications of self-attention and related ideas to image recognition [5, 34, 15, 14, 45, 46, 13, 1, 27], image synthesis [43, 26, 2], image captioning [39, 41, 4], and video prediction [17, 35].”
Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “Its ability to directly encode long-range interactions and its parallelizability, has led to state-of-the-art performance for various tasks [78,37,24,64,70,23].”
Axial Attention in Multidimensional Transformers (2019, background): “52 Image Transformer (Parmar et al., 2018) 3.”
Scaling Autoregressive Video Models (2019, methodology): “The approach of Parmar et al. (2018) uses overlapping 2D image blocks in each layer.”
Attention Augmented Convolutional Networks (2019, methodology): “In particular, the Image Transformer [32] extends the sinusoidal waves first introduced in the original Transformer [43] to 2 dimensional inputs and CoordConv [29] concatenates positional channels to an activation map.”
Flow++: Improving Flow-Based Generative Models with Variational Dequantization and Architecture Design (2019, methodology): “…models in the form of GANs (Goodfellow et al., 2014), and exact likelihood models like PixelRNN/CNN (van den Oord et al., 2016b;c), Image Transformer (Parmar et al., 2018), PixelSNAIL (Chen et al., 2017), NICE, RealNVP, and Glow (Dinh et al., 2014; 2016; Kingma & Dhariwal, 2018) – have recently…”
Waveglow: A Flow-based Generative Network for Speech Synthesis (2018, neutral): “As suggested in [1], and earlier work on likelihood-based generative models [14], we found that sampling z s from a Gaussian with a lower standard deviation from that assumed during training resulted in slightxly quality higher audio.”
Generating High Fidelity Images with Subscale Pixel Networks and Multidimensional Upscaling (2018, background): “…(van den Oord et al., 2016c) 3.83 3.57 Parallel Multiscale (Reed et al., 2017) 3.95 3.70 PixelSNAIL (Chen et al., 2017) 3.80 - Image Transformer (Parmar et al., 2018) 3 ) and P ( x d 1 : d 2 h , w , c | x d 1 : d 2 < , x : d 1 ) respectively. concatenated along the channel dimension into a…”
Glow: Generative Flow with Invertible 1x1 Convolutions (2018, positive): “Consistent with earlier work on likelihood-based generative models, we found that sampling from a reduced-temperature model (Parmar et al., 2018) often results in higher-quality samples.”