A Survey on Vision Transformer

By Kai Han, Yunhe Wang, Hanting Chen, Xinghao Chen, Jianyuan Guo, Zhenhua Liu, Yehui Tang, An Xiao, Chunjing Xu, Yixing Xu and 3 more. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020

3,751 citations recorded by the source corpus.

Transformer, first applied to the field of natural language processing, is a type of deep neural network mainly based on the self-attention mechanism. Thanks to its strong representation capabilities, researchers are looking at ways to apply transformer to computer vision tasks. In a variety of visual benchmarks, transformer-based models perform similar to or better than other types of networks such as convolutional and recurrent neural networks. Given its high performance and less need for vision-specific inductive bias, transformer is receiving more and more attention from the computer vision community. In this paper, we review these vision transformer models by categorizing them in different tasks and analyzing their advantages and disadvantages. The main categories we explore include the backbone network, high/mid-level vision, low-level vision, and video processing. We also include efficient transformer methods for pushing transformer into real device-based applications. Furthermore, we also take a brief look at the self-attention mechanism in computer vision, as it is the base component in transformer. Toward the end of this paper, we discuss the challenges and provide several further research directions for vision transformers.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d40c77c010c8dbef6142903a02f2a73a85012d5d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Chen Change Loy
4. Yunhe Wang
5. Qi Tian
6. Jun Liu
7. Yi Yang
8. Xiaogang Wang

Citations in the corpus

• Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “…NLP SC, 34 Vision-Language Intelligence: Tasks, Representation Learning, and Large Models [38] 2022 arXiv MM DC, 19 A survey on vision transformer [39] 2022 TPAMI CV DC, 23 Transformers in vision: A survey [40] 2021 CSUR CV SC, 38 A Survey of Visual Transformers [41] 2021 arXiv CV DC, 21 Video…”
• A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks (2023, background): “(Han et al., 2023) Computer Vision • Categorized vision transformer models based on different tasks • This existing paper analyzed transformer models’ advantages and disadvantages, and efficient transformer methods for the backbone network, while our survey categorizes transformer models based on ta”
• Vision GNN: An Image is Worth Graph of Nodes (2022, methodology): “Vision transformer was introduced for visual tasks from 2020 [13, 8, 3].”
• LLaVA-Prumerge: Adaptive Token Reduction for Efficient Large Multimodal Models (2024, neutral): “These components work together to improve the model’s capability to understand visual data [16].”
• Swin-UMamba: Mamba-based UNet with ImageNet-based pretraining (2024, neutral): “The impact of pretraining for the Mamba-based model in medical image segmentation tasks remains unclear, which has been proven to be quite effective for data-efficient medical image analysis with CNNs [6] and ViTs [7].”
• MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, background): “That is to say, the Transformer architectures address the long-range dependency modeling in CNNs, yet their computational complexity increases quadratically with the spatial dimen-sion [9].”
• Spike-driven Transformer (2023, background): “Incorporating the effectiveness of Transformer with the high energy efficiency of SNNs is a natural and exciting idea.”
• A Survey of Visual Transformers (2021, neutral): “[48] summarized the early visual Transformers and attention-based models.”
• Crossformer: Transformer Utilizing Cross-Dimension Dependency for Multivariate Time Series Forecasting (2023, background): “Readers can refer to the recent survey (Han et al., 2022) for comprehensive study on vision transformers.”
• Universal Multimodal Representation for Language Understanding (2023, background): “Most of previousworks focus on joint modeling images and texts, involving vision-language (VL) pre-training [15], [16], [17], [18], [19], [20] and multimodal (MM) application tasks [14], [21], [22], [23], [24].”
• Transformers in Time Series: A Survey (2022, background): “There are quite a few literature reviews from different aspects, such as in NLP applications [ Han et al. , 2021 ] , CV applications [ Han et al. , 2022], and efficient Transformers [Tay et al. , 2022].”
• Transformer in Transformer (2021, background): “Nowadays, transformer architectures have been used in a growing number of computer vision tasks [10] such as image recognition [6, 40, 29], object detection [46], and segmentation [43, 38].”
• VLP: A Survey on Vision-language Pre-training (2022, background): “Although there are many surveys on pre-trained language models[5, 6] and pre-trained vision models[7], to the best of our knowledge, this is the first survey focused on VLP.”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, neutral): “Recently, HierVST has significantly improved a voice style transfer performance of the E2E model; therefore, we conduct ablation studies by building up on HierVST.”
• Comparison of Vision Transformers and Convolutional Neural Networks in Medical Image Analysis: A Systematic Review (2024, neutral): “Unlike traditional convolutional approaches, ViTs treat an image as a sequence of patches and apply a transform model to these patches, enabling them to learn spatial hierarchies and relationships in the visual data [27, 28].”
• Recipe for a General, Powerful, Scalable Graph Transformer (2022, background): “Considering the great successes of Transformers in natural language processing (NLP) [52, 30] and recently also in computer vision [16, 23, 22], it is natural to study their applicability in the graph domain as well.”
• Multimodal Learning With Transformers: A Survey (2022, background): “While [50], [52], [53], [54] consider MML, their reviews are somewhat limited in the scope, taxonomy, and coverage.”