Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions

By Wenhai Wang, Enze Xie, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao. 2021

5,140 citations recorded by the source corpus.

Although convolutional neural networks (CNNs) have achieved great success in computer vision, this work investigates a simpler, convolution-free backbone network use-fid for many dense prediction tasks. Unlike the recently-proposed Vision Transformer (ViT) that was designed for image classification specifically, we introduce the Pyramid Vision Transformer (PVT), which overcomes the difficulties of porting Transformer to various dense prediction tasks. PVT has several merits compared to current state of the arts. (1) Different from ViT that typically yields low-resolution outputs and incurs high computational and memory costs, PVT not only can be trained on dense partitions of an image to achieve high output resolution, which is important for dense prediction, but also uses a progressive shrinking pyramid to reduce the computations of large feature maps. (2) PVT inherits the advantages of both CNN and Transformer, making it a unified backbone for various vision tasks without convolutions, where it can be used as a direct replacement for CNN backbones. (3) We validate PVT through extensive experiments, showing that it boosts the performance of many downstream tasks, including object detection, instance and semantic segmentation. For example, with a comparable number of parameters, PVT+RetinaNet achieves 40.4 AP on the COCO dataset, surpassing ResNet50+RetinNet (36.3 AP) by 4.1 absolute AP (see Figure 2). We hope that PVT could, serre as an alternative and useful backbone for pixel-level predictions and facilitate future research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Early Convolutions Help Transformers See Better (2021, background): “Dominant themes include multi-scale networks [14, 17, 25, 45, 50], increasing depth [42], and locality priors [5, 9, 17, 47, 49].”
VOLO: Vision Outlooker for Visual Recognition (2021, background): “Some of them [4, 19, 61, 54, 79] introduce local dependency into vision transformers by modifying the patch embedding block or the transformer block or both, while others [21, 37, 57] adopt a pyramid structure to reduce the overall computation while maintaining the models’ ability to capture low-lev”
Towards Robust Vision Transformer (2021, background): “Base on this seminal work, numerous variants have been proposed to improve ViTs from different perspectives containing training data efficiency [40], self-attention mechanism [25], introducing convolution [23,45,50] or pooling layers [20,43], etc.”
Multiview Transformers for Video Recognition (2022, background): “In practice, due to the limited amount of training data, applying similar multiscale processing in transformers [11, 21, 44, 74] to reduce complexity of the model has proven to be effective.”
Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Tasks (2021, methodology): “For image recognition, vision Transform-ers [9, 17, 45, 66, 68, 71, 73, 76] replace CNNs by an image patch tokenizer and a transformer encoder, which have been proved to obtain competitive performance as CNNs. [4, 5, 74] make attempts to apply Transformers on video recognition in a convolution-free…”
Scaling Vision Transformers (2021, background): “Pyramidal ViTs [45], designed for dense prediction tasks, follow a CNN-like pyramidal structure, that also reduces the size of the model.”
Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “Another concurrent work [60] explores a similar line of thinking to build multi-resolution feature maps on Transformers.”
Unified Contrastive Learning in Image-Text-Label Space (2022, methodology): “With this goal, numerous works have pushed the image recognition performance from different directions, such as data scale from MNIST [36] to ImageNet1K [10], model architectures from convolutional neural networks (CNNs) [23, 26, 34, 35, 40, 52, 53] to Transformers [15, 43, 57, 61, 64, 67, 72], and ”
Exploring Plain Vision Transformer Backbones for Object Detection (2022, background): “, Swin [40], MViT [16,32], PVT [53], and PiT [28].”
How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers (2021, background): “In [22, 13, 36, 38] the authors propose hierarchical versions of ViT.”
Few-Shot Class-Incremental Learning by Sampling Multi-Phase Tasks (2022, neutral): “, sequence transformer [83],vision transformer [84], [85], [86].”
Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “Recently, transformer architectures [7, 26, 86, 97, 104, 109] shine in semantic segmentation, focusing on multi-scale feature fusion [12, 84, 89, 101] and contextual feature aggregation [58, 79].”
Transformers in Vision: A Survey (2021, methodology): “Vision Transformer (ViT) [11] is the first work to showcase how Transformers can ‘altogether’ replace standard convolutions in deep neural networks on large-scale computer vision datasets.”
MLP-Mixer: An all-MLP Architecture for Vision (2021, background): “The Vision Transformers (ViTs) have model scales and patch resolutions similar to Mixer.”
SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformers (2021, methodology): “PVT shows considerable improvements over the ResNet counterpart on object detection and semantic segmentation.”
CvT: Introducing Convolutions to Vision Transformers (2021, methodology): “With fewer paramerters, CvT-13 achieves a 81.6% ImageNet Top-1 accuracy, outperforming PVT-Small [34], T2T-ViT t -14 [41], TNT-S [14] by 1.7%, 0.8%, 0.2% respectively.”
ViViT: A Video Vision Transformer (2021, background): “ViT has inspired a large amount of follow-up work in the community, and we note that there are a number of concurrent approaches on extending it to other tasks in computer vision [71, 74, 84, 85] and improving its data-efﬁciency [64, 48].”
Transformer in Transformer (2021, methodology): “For fair comparison, we adopt the training and testing settings in PVT [36] and add a 2×2 average pooling to make the output size of TNT backbone the same as that of PVT and ResNet.”