Visual Transformers: Token-based Image Representation and Processing for Computer Vision

By Bichen Wu, Chenfeng Xu, Xiaoliang Dai, Alvin Wan, Peizhao Zhang, Masayoshi Tomizuka, Kurt Keutzer, Péter Vajda. 2020

755 citations recorded by the source corpus.

Computer vision has achieved great success using standardized image representations -- pixel arrays, and the corresponding deep learning operators -- convolutions. In this work, we challenge this paradigm: we instead (a) represent images as a set of visual tokens and (b) apply visual transformers to find relationships between visual semantic concepts. Given an input image, we dynamically extract a set of visual tokens from the image to obtain a compact representation for high-level semantics. We then use visual transformers to operate over the visual tokens to densely model relationships between them. We find that this paradigm of token-based image representation and processing drastically outperforms its convolutional counterparts on image classification and semantic segmentation. To demonstrate the power of this approach on ImageNet classification, we use ResNet as a convenient baseline and use visual transformers to replace the last stage of convolutions. This reduces the stage's MACs by up to 6.9x, while attaining up to 4.53 points higher top-1 accuracy. For semantic segmentation, we use a visual-transformer-based FPN (VT-FPN) module to replace a convolution-based FPN, saving 6.5x fewer MACs while achieving up to 0.35 points higher mIoU on LIP and COCO-stuff.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, methodology): “…e.g. for object detection (Hu et al., 2018; Carion et al., 2020), video processing (Wang et al., 2018; Sun et al., 2019), image classiﬁcation (Wu et al., 2020), unsupervised object discovery (Locatello et al., 2020), or uniﬁed text-vision tasks (Chen et al., 2020c; Lu et al., 2019; Li et…”
VOLO: Vision Outlooker for Visual Recognition (2021, background): “including image classification [2], [3], [4], object detection [5], [6] and image processing like denoising, super-resolution and deraining [7].”
Perceiver: General Perception with Iterative Attention (2021, methodology): “A variety of strategies have been proposed to reduce the size of the input to the Transformer so it can be used on domains that are otherwise too large, including subsampling the input (Chen et al., 2020a) or by ﬁrst preprocessing the input using convolutions (e.g. Wu et al. 2020).”
Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet (2021, methodology): “Self-attention models for language modeling like Transformers [37] have been recently applied to vision tasks, including image classification [5, 12, 43], object detec-”
Going deeper with Image Transformers (2021, methodology): “On ImageNet1k-val [55], our model is on par with the state of the art (86.5”
Multiscale Vision Transformers (2021, background): “A separate line of works attempts at modeling visual data with learnt discretized token sequences [104, 83, 14, 109, 18].”
LambdaNetworks: Modeling Long-Range Interactions Without Attention (2021, background): “These include image classification (Bello et al., 2019; Ramachandran et al., 2019; Cordonnier et al., 2019; Zhao et al., 2020; Wu et al., 2020; Dosovitskiy et al., 2020); object detection and object-centric tasks (Wang et al., 2018; Hu et al., 2018a; Carion et al., 2020; Locatello et al., 2020);…”
ConViT: improving vision transformers with soft convolutional inductive biases (2021, methodology): “A breadth of approaches have been taken to imbue CNN architectures with nonlocality [34, 35, 36, 28].”
LeViT: a Vision Transformer in ConvNet’s Clothing for Faster Inference (2021, background): “Bottleneck transformers [46] and “Visual Transformers” [47] (not to be confused with ViT) are both generic architectures that can also be used for detection and object segmentation.”
Do Vision Transformers See Like Convolutional Neural Networks? (2021, background): “Prior works have looked at local multiheaded self-attention, drawing from the structure of convolutional receptive fields [30, 36], directly combining CNNs with self-attention [4, 2, 46] or applying Transformers to smaller-size images [6, 9].”
Multiattention Network for Semantic Segmentation of Fine-Resolution Remote Sensing Images (2020, background): “Meanwhile, another burgeoning research area for semantic segmentation is how to embed the dot-product attention into a Graph Convolutional Network (GCN) and optimize the complexity of the attention [79-83].”
MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (2022, methodology): “In high-level vision, Transformer has been widely applied in image classification [7, 19, 25, 27, 49, 65], object detection [6, 22, 23, 60, 86, 93], semantic segmentation [16, 69,76,77,90,91], human pose estimation [14,15,32,37,42, 43,53,79,89], etc .”
A Survey of Visual Transformers (2021, methodology): “VTs [52] decouples semantic concepts for the input image into different channels and relates them densely through the encoder block, namely VT-block.”
Efficient Long-Range Attention Network for Image Super-resolution (2022, background): “The SA mechanism in transformers can effectively model the dependency across data, and it has achieved impressive results on several high-level vision tasks, such as image classification [13, 26, 34, 35, 44, 52], image detection [8, 32, 35, 52], and segmentation [7, 35, 57, 63].”
SwinIR: Image Restoration Using Swin Transformer (2021, methodology): “When used in vision problems such as image classification [66, 19, 84, 56, 45, 55, 75], object detection [6, 53, 74, 56], segmentation [84, 99, 56, 4] and crowd counting [47, 69], it learns to attend to important image regions by exploring the global interactions between different regions.”
A Survey on Vision Transformer (2020, methodology): “[240] adopted ResNet as a convenient baseline and used vision transformers to replace the last stage of convolutions.”