A Survey of Transformers

By Tianyang Lin, Yuxin Wang, Xiangyang Liu, Xipeng Qiu. 2021

1,529 citations recorded by the source corpus.

Transformers have achieved great success in many artificial intelligence fields, such as natural language processing, computer vision, and audio processing. Therefore, it is natural to attract lots of interest from academic and industry researchers. Up to the present, a great variety of Transformer variants (a.k.a. X-formers) have been proposed, however, a systematic and comprehensive literature review on these Transformer variants is still missing. In this survey, we provide a comprehensive review of various X-formers. We first briefly introduce the vanilla Transformer and then propose a new taxonomy of X-formers. Next, we introduce the various X-formers from three perspectives: architectural modification, pre-training, and applications. Finally, we outline some potential directions for future research.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Pre-Trained Models: Past, Present and Future (2021, background): “For more details of Transformer, please refer to its original paper (Vaswani et al., 2017) and the survey paper (Lin et al., 2021).”
Pre-trained models for natural language processing: A survey (2020, background): “A successful implementation of such an idea is the Transformer [178], which adopts the fully-connected self-attention architecture as well as other useful designs, such as positional embeddings, layer normalization, and residual connections.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “attention and linearized self-attention are proposed to reduce its complexity and hence extend T-PTLMs to long input sequences also [206].”
Comparison of Vision Transformers and Convolutional Neural Networks in Medical Image Analysis: A Systematic Review (2024, methodology): “The softmax function can be applied to the scaled dot products of queries with keys to provide a distribution of weights [49].”
Transformer-Based Visual Segmentation: A Survey (2023, background): “We believe it would be beneﬁcial for the community to summarize these works and keep tracking this evolving ﬁeld.”
Transformers in Time-Series Analysis: A Tutorial (2022, background): “Recently, Transformers have been successfully employed to tackle various problems in machine learning and achieve state-of-the-art performance [2].”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “There are a large number of variants that improve the basic Transformer architecture, and the readers can refer to the related surveys [158], [159] for a comprehensive discussion.”
PhysFormer: Facial Video-based Physiological Measurement with Temporal Difference Transformer (2021, positive): “Recently, due to its excellent long-range attentional modeling capacities in solving sequence-to-sequence issues, transformer [21,31] has been successfully applied in many artificial intelligence tasks such as natural language processing (NLP) [55], image [14] and video [3] analysis.”
A Survey of Visual Transformers (2021, background): “, which provides a systematic review of various Transformers, but they only mention vision applications sketchily [49].”
Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content? (2022, background): “Large-scale transformer-based language models revolutionized many NLP tasks [Lin et al. 2021].”
ChatGPT: Jack of all trades, master of none (2023, background): “Only 18% of users using the OpenAI API queried GPT-3 model with tasks familiar to typical NLP tasks, most of which are analytical.”
A Review of the Role of Artificial Intelligence in Healthcare (2023, background): “Transformer Transformer is a critical deep learning model and is broadly used in various areas, namely, computer vision (CV), natural language processing (NLP), and speech processing [31].”
Multimodal Learning With Transformers: A Survey (2022, background): “recently introduced, with a range of emphases including general Transformers [48], efficient designs [49], visualization [50], computer vision tasks [51], [52], [53], [54], medical imag-”
ChatGPT is not all you need. A State of the Art Review of large Generative AI models (2023, background): “Despite being fed with an enormous size of data, thanks to the rise of computing we can design deep neural networks [18], transformers [22] and other models such as generative adversarial networks [9] or variational autoencoders [23] whose capacity is able to model the complexity of the data,…”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, background): “Instead, our multimodal prompt interface allows us to harness the latest advances in large transformer models (Lin et al., 2021; Tay et al., 2020; Khan et al., 2021) for developing scalable multi-task robot learners.”