ResT: An Efficient Transformer for Visual Recognition

By Qing-Long Zhang, Yubing Yang. 2021

304 citations recorded by the source corpus.

This paper presents an efficient multi-scale vision Transformer, called ResT, that capably served as a general-purpose backbone for image recognition. Unlike existing Transformer methods, which employ standard Transformer blocks to tackle raw images with a fixed resolution, our ResT have several advantages: (1) A memory-efficient multi-head self-attention is built, which compresses the memory by a simple depth-wise convolution, and projects the interaction across the attention-heads dimension while keeping the diversity ability of multi-heads; (2) Position encoding is constructed as spatial attention, which is more flexible and can tackle with input images of arbitrary size without interpolation or fine-tune; (3) Instead of the straightforward tokenization at the beginning of each stage, we design the patch embedding as a stack of overlapping convolution operation with stride on the 2D-reshaped token map. We comprehensively validate ResT on image classification and downstream tasks. Experimental results show that the proposed ResT can outperform the recently state-of-the-art backbones by a large margin, demonstrating the potential of ResT as strong backbones. The code and models will be made publicly available at https://github.com/wofmanaf/ResT.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Transformers in Vision: A Survey (2021, background): “ResT [110] is another hierarchical architecture which applies a CNN block at the beginning for patch-embedding.”
UNetFormer: A UNet-like transformer for efficient semantic segmentation of remote sensing urban scene imagery (2021, unknown): “…the computational complexity of the Transformer-based encoder is much higher than the CNN-based encoder due to its square-complexity self-attention mechanism (Vaswani et al., 2017), which seriously affects its potential and feasibility for urban-related (Liu et al., 2021; Zhang and Yang, 2021).”
Self-Training Multi-Sequence Learning with Transformer for Weakly Supervised Video Anomaly Detection (2022, background): “Because transformer lacks the ability of local perception, many works combine convolution and transformer (d’Ascoli et al. 2021; Wu et al. 2021; Li et al. 2021; Xu et al. 2021; Yan et al. 2021; Zhang and Yang 2021; Liu et al. 2021a).”
CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention (2021, background): “The competitors are all competitive vision transformers, including DeiT (Touvron et al., 2021), PVT (Wang et al., 2021), T2T-ViT (Yuan et al., 2021), TNT (Han et al., 2021), CViT (Chen et al., 2021a), Twins (Chu et al., 2021), Swin (Liu et al., 2021b), NesT (Zhang et al., 2021b), CvT (Wu et al., 202”