A Survey of Visual Transformers

By Yang Liu, Yao Zhang, Yixin Wang, Feng Hou, Jin Yuan, Jiang Tian, Yang Zhang, Zhongchao Shi, Jianping Fan, Zhiqiang He. IEEE Transactions on Neural Networks and Learning Systems, 2021

550 citations recorded by the source corpus.

Transformer, an attention-based encoder–decoder model, has already revolutionized the field of natural language processing (NLP). Inspired by such significant achievements, some pioneering works have recently been done on employing Transformer-liked architectures in the computer vision (CV) field, which have demonstrated their effectiveness on three fundamental CV tasks (classification, detection, and segmentation) as well as multiple sensory data stream (images, point clouds, and vision-language data). Because of their competitive modeling capabilities, the visual Transformers have achieved impressive performance improvements over multiple benchmarks as compared with modern convolution neural networks (CNNs). In this survey, we have reviewed over 100 of different visual Transformers comprehensively according to three fundamental CV tasks and different data stream types, where taxonomy is proposed to organize the representative methods according to their motivations, structures, and application scenarios. Because of their differences on training settings and dedicated vision tasks, we have also evaluated and compared all these existing visual Transformers under different configurations. Furthermore, we have revealed a series of essential but unexploited aspects that may empower such visual Transformers to stand out from numerous architectures, e.g., slack high-level semantic embeddings to bridge the gap between the visual Transformers and the sequential ones. Finally, two promising research directions are suggested for future investment. We will continue to update the latest articles and their released source codes at https://github.com/liuyang-ict/awesome-visual-transformers.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/333212e246fb65f7c9d43862021e78f007c48449

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Fine-tuning Image Transformers using Learnable Memory (2022, background): “We refer to [20, 27] for an in-depth survey of recent literature.”
Multimodal Learning With Transformers: A Survey (2022, background): “recently introduced, with a range of emphases including general Transformers [48], efficient designs [49], visualization [50], computer vision tasks [51], [52], [53], [54], medical imag-”
Large-scale Multi-modal Pre-trained Models: A Comprehensive Survey (2023, background): “…A survey on vision transformer [39] 2022 TPAMI CV DC, 23 Transformers in vision: A survey [40] 2021 CSUR CV SC, 38 A Survey of Visual Transformers [41] 2021 arXiv CV DC, 21 Video Transformers: A Survey [42] 2022 arXiv CV DC, 24 Threats to Pre-trained Language Models: Survey and Taxonomy [43] 2022…”