Rethinking Spatial Dimensions of Vision Transformers

By Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, Seong Joon Oh. 2021

757 citations recorded by the source corpus.

Vision Transformer (ViT) extends the application range of transformers from language processing to computer vision tasks as being an alternative architecture against the existing convolutional neural networks (CNN). Since the transformer-based architecture has been innovative for computer vision modeling, the design convention towards an effective architecture has been less studied yet. From the successful design principles of CNN, we investigate the role of spatial dimension conversion and its effectiveness on transformer-based architecture. We particularly attend to the dimension reduction principle of CNNs; as the depth increases, a conventional CNN increases channel dimension and decreases spatial dimensions. We empirically show that such a spatial dimension reduction is beneficial to a transformer architecture as well, and propose a novel Pooling-based Vision Transformer (PiT) upon the original ViT model. We show that PiT achieves the improved model capability and generalization performance against ViT. Throughout the extensive experiments, we further show PiT outperforms the baseline on several tasks such as image classification, object detection, and robustness evaluation. Source codes and ImageNet models are available at https://github.com/naver-ai/pit.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Towards Robust Vision Transformer (2021, background): “PVT [43], PiT [20] and Swin [25] employ this design principle into ViTs.”
Exploring Plain Vision Transformer Backbones for Object Detection (2022, background): “, Swin [40], MViT [16,32], PVT [53], and PiT [28].”
P2T: Pyramid Pooling Transformer for Scene Understanding (2021, background): “Unfortunately, most of recent development on transformer focuses on improving image classification on ImageNet [1] without considering this problem [19]–[22].”
Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformer (2021, methodology): “The ﬁrst pipeline, as shown in the second pathway, is to perform structured compression based on local spatial prior, such as local linear projection (Wang et al. 2021a), convolutional projection (Heo et al. 2021), and shift windows (Liu et al. 2021).”
Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet (2021, methodology): “…into two stages, Token-to-Token ViT [13] aggregates neighboring tokens throughout the network, CrossViT [1] processes patches at two scales, PiT [6] adds pooling layers, LeViT [5] uses convolutional embeddings and modiﬁed attention/normalization layers, CvT [12] uses depthwise convolutions in…”
Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations (2022, methodology): “…also some attempts to accelerate the computation of the Transformer model, including proposing more efficient attention mechanisms (Wang et al., 2020a; Kitaev et al., 2020; Choromanski et al., 2020) and the compressed Transformer structures (Liu et al., 2021; Heo et al., 2021; Wang et al., 2021a).”
A Survey on Vision Transformer (2020, methodology): “transformer models [232], [148], [209], [61], [279], [167] including PVT [232], HVT [168], Swin Transformer [148] and PiT [92].”
Uformer: A General U-Shaped Transformer for Image Restoration (2021, background): “Some works [19, 59] focus on establishing a pyramid Transformer architecture simlilar to ConvNet-based structure.”
MobileViT: Light-weight, General-purpose, and Mobile-friendly Vision Transformer (2021, methodology): “, 2021), PiT (Heo et al., 2021), ConViT (d’Ascoli et al.”
ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Bias (2021, background): “[79, 27] design sequential stage-wise structures while [46, 32] apply attention within local windows.”
MedViT: A Robust Vision Transformer for Generalized Medical Image Classification (2023, background): “PiT [29] follows a similar pyramid structure as”
Patch-level Representation Learning for Self-supervised Vision Transformers (2022, methodology): “Inspired by the success of the Vision Transformers, a number of variants [16, 21, 28, 30, 38, 40, 47] have been developed.”
Vision Transformer for Small-Size Datasets (2021, background): “For example, some adopted a hierarchical structure like CNNs to leverage various receptive fields [12, 24, 39], and the others tried to modify the self-attention mechanism itself [22,24,34,39,40].”
All Tokens Matter: Token Labeling for Training Better Vision Transformers (2021, background): “Moreover, there are also some works [20, 25, 40] adopting a pyramid structure to reduce the overall computation while maintaining the model’s ability to capture low-level features.”
DaViT: Dual Attention Vision Transformers (2022, background): “Similarly, follow-up works [25,49,56,47,59,20,73,43,44] model global relationships on the patch-level tokens.”