Transformers in Vision: A Survey

By Salman Hameed Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, Mubarak Shah. 2021

3,683 citations recorded by the source corpus.

Astounding results from Transformer models on natural language tasks have intrigued the vision community to study their application to computer vision problems. Among their salient benefits, Transformers enable modeling long dependencies between input sequence elements and support parallel processing of sequence as compared to recurrent networks, e.g., Long short-term memory. Different from convolutional networks, Transformers require minimal inductive biases for their design and are naturally suited as set-functions. Furthermore, the straightforward design of Transformers allows processing multiple modalities (e.g., images, videos, text, and speech) using similar processing blocks and demonstrates excellent scalability to very large capacity networks and huge datasets. These strengths have led to exciting progress on a number of vision tasks using Transformer networks. This survey aims to provide a comprehensive overview of the Transformer models in the computer vision discipline. We start with an introduction to fundamental concepts behind the success of Transformers, i.e., self-attention, large-scale pre-training, and bidirectional feature encoding. We then cover extensive applications of transformers in vision including popular recognition tasks (e.g., image classification, object detection, action recognition, and segmentation), generative modeling, multi-modal tasks (e.g., visual-question answering, visual reasoning, and visual grounding), video processing (e.g., activity recognition, video forecasting), low-level vision (e.g., image super-resolution, image enhancement, and colorization), and three-dimensional analysis (e.g., point cloud classification and segmentation). We compare the respective advantages and limitations of popular techniques both in terms of architectural design and their experimental value. Finally, we provide an analysis on open research directions and possible future works. We hope this effort will ignite further interest in the community to solve current challenges toward the application of transformer models in computer vision.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, background): “Recent years have seen a rapid progress made in vision-language pretraining (Uppal et al., 2020; Han et al., 2021; Khan et al., 2021).”
SimpleClick: Interactive Image Segmentation with Simple Vision Transformers (2022, neutral): “Recently, ViT-based approaches [17, 24, 44, 46, 50] have shown competitive performance on segmentation tasks compared to ConvNets.”
Dynamic Neural Networks: A Survey (2021, background): “Readers who are interested in this topic may refer to review studies [139], [140], [141].”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Transformer-Based Architecture [317] has proven effective for both language and vision domains, and become the dominant choice for state-of-the-art foundation models.”
Online Continual Learning with Contrastive Vision Transformer (2022, background): “Since then, a lot of effort has been dedicated to improving Vision Transformers’ data efficiency and model efficiency [92,32,91,40], where an effective direction is to strategically integrate properties of convolution into the Transformer architecture [73,91,48,84,26,94,23,17].”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, background): “W ITH the rapid development of supervised learning, current learning systems have achieved or even surpassed humanlevel performances in many tasks [1], [2], [3], [4], [5], [6].”
CoAtNet: Marrying Convolution and Attention for All Data Sizes (2021, background): “For a more comprehensive review of vision Transformers, we refer readers to the dedicated surveys [36, 37].”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, background): “We refer the readers to [37, 31, 37] for comprehensive surveys.”
A Time Series is Worth 64 Words: Long-term Forecasting with Transformers (2022, background): “…has achieved great success on various application ﬁelds such as natural language processing (NLP) (Kalyan et al., 2021), computer vision (CV) (Khan et al., 2021), speech (Karita et al., 2019), and more recently time series (Wen et al., 2022), beneﬁting from its attention mechanism which can…”
Handwriting Transformers (2021, background): “Its worth mentioning that a CNN-only design would struggle to model long-term relations within sequences while an architecture based solely on transformer networks would demand large amount of data and longer training times [16].”
Human Action Recognition From Various Data Modalities: A Review (2020, methodology): “This design enables Transformer to perform well concerning long-term dependency modeling, multi-modal fusion, and multi-task processing [172], [173].”
A Practical Survey on Faster and Lighter Transformers (2021, background): “The self-attention’s main advantage is its ability to relate any two positions of the input sequence regardless of their distance, thus increasing performance signiicantly on a wide range of tasks, including natural language processing (NLP) [10, 24, 121], computer vision [12, 27, 57], speech recogn”
Deep Neural Networks and Tabular Data: A Survey (2021, background): “Inspired by the recent surge of interest in transformer-based methods and their successes on text and visual data [75], [111], researchers and practitioners have proposed multiple approaches using deep attention mechanisms [4] for heterogeneous tabular data.”
Point-BERT: Pre-training 3D Point Cloud Transformers with Masked Point Modeling (2021, methodology): “It has intrigued various vision tasks [12, 19], such as object classification [6, 9], detection [4, 66] and segmentation [51, 64].”
Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks (2021, methodology): “Readers are referred to recent surveys [45], [46] for a more comprehensive reviewof the use of transformermethods for visual tasks.”
Spectral–Spatial Morphological Attention Transformer for Hyperspectral Image Classification (2023, methodology): “As cutting-edge backbone networks, transformers utilize self-attention techniques to process and analyze sequential data more efficiently [44].”
Curriculum Learning: A Survey (2021, background): “Curriculum learning in computer vision The current focus of the computer vision researchers is the development of vision transformers (Carion et al. 2020; Caron et al. 2021; Dosovitskiy et al. 2021; Jaegle et al. 2021; Khan et al. 2021; Wu et al. 2021; Zhu et al. 2020), which make use of the global ”
MaxViT: Multi-Axis Vision Transformer (2022, neutral): “We refer readers to dedicated surveys [42,42] of vision Transformers for a complete review.”