VOLO: Vision Outlooker for Visual Recognition

By Li Yuan, Qibin Hou, Zihang Jiang, Jiashi Feng, Shuicheng Yan. 2021

408 citations recorded by the source corpus.

Recently, Vision Transformers (ViTs) have been broadly explored in visual recognition. With low efficiency in encoding fine-level features, the performance of ViTs is still inferior to the state-of-the-art CNNs when trained from scratch on a midsize dataset like ImageNet. Through experimental analysis, we find it is because of two reasons: 1) the simple tokenization of input images fails to model the important local structure such as edges and lines, leading to low training sample efficiency; 2) the redundant attention backbone design of ViTs leads to limited feature richness for fixed computation budgets and limited training samples. To overcome such limitations, we present a new simple and generic architecture, termed Vision Outlooker (VOLO), which implements a novel outlook attention operation that dynamically conduct the local feature aggregation mechanism in a sliding window manner across the input image. Unlike self-attention that focuses on modeling global dependencies of local features at a coarse level, our outlook attention targets at encoding finer-level features, which is critical for recognition but ignored by self-attention. Outlook attention breaks the bottleneck of self-attention whose computation cost scales quadratically with the input spatial dimension, and thus is much more memory efficient. Compared to our Tokens-To-Token Vision Transformer (T2T-ViT), VOLO can more efficiently encode fine-level features that are essential for high-performance visual recognition. Experiments show that with only 26.6 M learnable parameters, VOLO achieves 84.2% top-1 accuracy on ImageNet-1 K without using extra training data, 2.7% better than T2T-ViT with a comparable number of parameters. When the model size is scaled up to 296 M parameters, its performance can be further improved to 87.1%, setting a new record for ImageNet-1 K classification. In addition, we also take the proposed VOLO as pretrained models and report superior performance on downstream tasks, such as semantic segmentation. Code is available at https://github.com/sail-sg/volo.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Masked Autoencoders Are Scalable Vision Learners (2021, neutral): “1% (512 size) [56], based on advanced networks.”
Combined Scaling for Zero-shot Transfer Learning (2021, methodology): “…models have an accuracy around 76% on ImageNet, which is only comparable with a supervised ResNet-50 (He et al., 2015), and signiﬁcantly worse than the state-of-the-art supervised training on ImageNet (without extra data: 87.1% (Yuan et al., 2021), and with extra data: 90.88% (Dai et al., 2021)).”
Swin Transformer V2: Scaling Up Capacity and Resolution (2021, background): “Since these works, numerous vision Transformer variants have been proposed to improve the accuracy at relatively small scale [8, 14, 23, 31, 50, 55, 58, 62, 64, 65, 67].”
Spikformer: When Spiking Neural Network Meets Transformer (2022, background): “…tasks in computer vision, including image classification (Dosovitskiy et al., 2020; Yuan et al., 2021a), object detection (Carion et al., 2020; Zhu et al., 2020; Liu et al., 2021), semantic segmentation (Wang et al., 2021; Yuan et al., 2021b) and low-level image processing (Chen et al., 2021).”
Combined Scaling for Open-Vocabulary Image Classiﬁcation (2022, methodology): “…models have an accuracy around 76% on ImageNet, which is only comparable with a supervised ResNet-50 (He et al., 2015), and signiﬁcantly worse than the state-of-the-art supervised training on ImageNet (without extra data: 87.1% (Yuan et al., 2021), and with extra data: 90.88% (Dai et al., 2021)).”
Neighborhood Attention Transformer (2022, background): “Some works focused on data efficiency, with minor changes in the architecture [27, 35, 11, 28, 10], while others focused on efficiency and transferrability to downstream tasks [20, 36, 29, 10].”
MambaOut: Do We Really Need Mamba for Vision?* (2024, neutral): “…under the spirit of RNN. Neverthe-less, their experiments show that the SSM based models for vision, in reality, lead to underwhelming performance compared with state-of-the-art convolutional [23, 31, 39, 54, 58, 70, 86, 98, 100, 101] and attention-based models [19, 24, 51, 76, 80, 81, 100, 103].”
CDTrans: Cross-domain Transformer for Unsupervised Domain Adaptation (2021, background): “2021), semantic segmentation (Yuan et al. 2021b) and object ReID (He et al.”
VC-GPT: Visual Conditioned GPT for End-to-End Generative Vision-and-Language Pre-training (2022, result): “A similar ﬁnding is also reported in several previous works (Kim et al., 2021; Yuan et al., 2021).”
DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition (2023, methodology): “…training data [21], [46] and strong data augmentation strategies [22], [47], [48], [49], [50], [51], Transformer-based methods [25], [52], [53], [54], [55], [56], [57], [58] achieve exciting performance improvements on various vision tasks, i.e., image classiﬁcation [22], [26], [41], [44],…”
An Empirical Study of Training End-to-End Vision-and-Language Transformers (2021, methodology): “In this work, we compare the original ViT [10], DeiT [45], Distilled-DeiT [45], CaiT [46], VOLO [56], BEiT [2], Swin Transformer [30] and CLIP-ViT [35], to provide a comprehensive analysis on the role of vision transformers.”
Scaling Up Your Kernels to 31×31: Revisiting Large Kernel Design in CNNs (2022, background): “been greatly challenged by Vision Transformers (ViTs) [34, 59, 86, 94], which have shown leading performances on many visual tasks – not only image classification [34, 104] and representation learning [4, 9, 16, 100], but also many downstream tasks such as object detection [24, 59], semantic segment”
Masked Autoencoders for Point Cloud Self-supervised Learning (2022, background): “Since ViT [12], Trans-formers architectures have been popular in computer vision [55,25,56,43,42,59,16,6].”
Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition (2021, background): “In addition, our Vision Permutator encodes not only coarse-level token representations (with 16 16 tokens) but also features at fine-level (with 32 32 tokens), which has been demonstrated important in vision transformers [46].”
Transformers in Medical Image Analysis: A Review (2022, methodology): “[48] proposed the Vision Outlooker (VOLO), a ViT model that replaced the original attention mechanism with the outlooker attention [72].”
FLatten Transformer: Vision Transformer using Focused Linear Attention (2023, background): “Similar insights have been adopted in the following researches [55, 54] and also extend to downstream tasks [22].”
UniFormer: Unifying Convolution and Self-Attention for Visual Recognition (2022, methodology): “block number, channel number and head dimension are [2, 5, 8, 2], [56, 112, 224, 448] and 28.”
MViTv2: Improved Multiscale Vision Transformers for Classification and Detection (2021, background): “Table 2 shows our MViT and state-of-the-art CNNs and Transformers (without external data or distillation models [43, 75, 87]).”