DeepViT: Towards Deeper Vision Transformer

By Daquan Zhou, Bingyi Kang, Xiaojie Jin, Linjie Yang, Xiaochen Lian, Qibin Hou, Jiashi Feng. 2021

638 citations recorded by the source corpus.

Vision transformers (ViTs) have been successfully applied in image classification tasks recently. In this paper, we show that, unlike convolution neural networks (CNNs)that can be improved by stacking more convolutional layers, the performance of ViTs saturate fast when scaled to be deeper. More specifically, we empirically observe that such scaling difficulty is caused by the attention collapse issue: as the transformer goes deeper, the attention maps gradually become similar and even much the same after certain layers. In other words, the feature maps tend to be identical in the top layers of deep ViT models. This fact demonstrates that in deeper layers of ViTs, the self-attention mechanism fails to learn effective concepts for representation learning and hinders the model from getting expected performance gain. Based on above observation, we propose a simple yet effective method, named Re-attention, to re-generate the attention maps to increase their diversity at different layers with negligible computation and memory cost. The pro-posed method makes it feasible to train deeper ViT models with consistent performance improvements via minor modification to existing ViT models. Notably, when training a deep ViT model with 32 transformer blocks, the Top-1 classification accuracy can be improved by 1.6% on ImageNet. Code is publicly available at https://github.com/zhoudaquan/dvit_repo.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Neurology, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Neurology

In Model Architectures

Citations in the corpus

PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel (2023, background): “• DeepViT [35]: model size 8B, and batch size 105 for 2 nodes and 120 for 4 nodes.”
VOLO: Vision Outlooker for Visual Recognition (2021, background): “There are also some works [78, 71, 52, 18] aiming at solving the optimization and scaling problems of ViTs.”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “Transformers have enabled a uniﬁed architecture paradigm for different modalities, which only need the modality-speciﬁc tokenizers to convert inputs from different modalities into uniﬁed input token sequences.”
Scaling Vision Transformers (2021, background): “Recently, [41, 53] explore strategies to scale the depth of ViTs.”
Transformers in Vision: A Survey (2021, background): “DeepViT [253] observes that the similarity between attention maps of deeper layer is high and hinders scaling models depth.”
CoAtNet: Marrying Convolution and Attention for All Data Sizes (2021, methodology): “Subsequent works use special regularization and stronger data augmentation to improve the vanilla ViT [16, 17, 18], yet none of these ViT variants could outperform the SOTA convolution-only models on ImageNet classification given the same amount of data and computation [19, 20].”
Focal Self-attention for Local-Global Interactions in Vision Transformers (2021, methodology): “Even without multi-scale design as in CNNs, a number of Transformer-based works such as TNT [32], DeepViT [89] and CaiT [58] achieve superior performance to CNN models with comparable model size and computational cost.”
Attention mechanisms in computer vision: A survey (2021, positive): “Following ViT, many transformer-based architectures such as PCT [27], IPT [79], T2T-ViT [44], DeepViT [166], SETR [81], PVT [45], CaiT [167], TNT [82], Swintransformer [46], Query2Label [83], MoCoV3 [84], BEiT [85], SegFormer [86], FuseFormer [168] and MAE [169] have appeared, with excellent results”
Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition (2021, background): “Since then, a great number of transformer-based classification models appear, aiming at advancing the original vision transformer by either introducing locality [9], [25], [26], [27], [28], [29], or scaling the depth [8], [30], or tailoring powerful optimization strategies [7].”
EfficientFormer: Vision Transformers at MobileNet Speed (2022, background): “Despite the success in most vision tasks, ViT-based models cannot compete with the well-studied lightweight CNNs [21, 49] when the inference speed is the major concern [50, 51, 52], especially on resource-constrained edge devices [17].”
MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer (2021, methodology): “Comparison with CNNs. Figure 6a shows that MobileViT outperforms light-weight CNNs across different network sizes (MobileNetv1 (Howard et al., 2017 et al., 2021b), PVT (Wang et al., 2021), CAIT (Touvron et al., 2021b), DeepViT (Zhou et al., 2021), CeiT (Yuan et al., 2021a), CrossViT (Chen et al., 20”
DaViT: Dual Attention Vision Transformers (2022, background): “Hierarchical designs are widely adopted to transformers [37,56,58,16,75,52,40,69,33,1,31,77,48,52,8,30,67,28,64,32,78] in vision.”
HiViT: Hierarchical Vision Transformer Meets Masked Image Modeling (2022, background): “Recently, the computer vision community has witnessed the emerge of vision transformers [14, 30, 47, 60, 10, 27], transplanted from the language models [46, 12], that replaced the dominance of convolutional neural networks [25, 23, 39].”
SwinTrack: A Simple and Strong Baseline for Transformer Tracking (2021, background): “With the success of the first applicable convolution-free network architecture and a vision of a shared pre-trained backbone network for CV and NLP tasks, a family of ViT variants was proposed [38] [28] [6] [47].”
A Simple Single-Scale Vision Transformer for Object Localization and Instance Segmentation (2021, background): “Attention scores of ViTs are also found to gradually become indistinguishable as the encoder goes deeper, leading to identical and redundant feature maps, and plateaued performance [47].”
Not All Images are Worth 16x16 Words: Dynamic Transformers for Efficient Image Recognition (2021, methodology): “Some following works such as T2T-ViT [55], TNT [17], CaiT [39], DeepViT [62], CPVT [6], LocalViT [28] and CrossViT [5] focus on improving the architecture design of ViT.”
VMamba: Visual State Space Model (2024, background): “Early ViT-based models usually require large-scale datasest [12] and appear in a plain configuration [62, 67, 2, 35].”
Vision Transformer with Deformable Attention (2022, methodology): “In the meantime, the observations in [3,52] have revealed that different queries have similar attention maps in visual attention models.”
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training (2022, background): “The vision transformer [21] also improves a series of computer vision tasks including image classification [67, 89], object detection [8, 38], semantic segmentation [81], object tracking [14, 17], and video recognition [6, 3].”