PVT v2: Improved baselines with Pyramid Vision Transformer

By Wenhai Wang, Enze Xie, Xiang-Yang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, Ling Shao. 2021

2,512 citations recorded by the source corpus.

Transformers have recently lead to encouraging progress in computer vision. In this work, we present new baselines by improving the original Pyramid Vision Transformer (PVT v1) by adding three designs: (i) a linear complexity attention layer, (ii) an overlapping patch embedding, and (iii) a convolutional feed-forward network. With these modifications, PVT v2 reduces the computational complexity of PVT v1 to linearity and provides significant improvements on fundamental vision tasks such as classification, detection, and segmentation. In particular, PVT v2 achieves comparable or better performance than recent work such as the Swin transformer. We hope this work will facilitate state-of-the-art transformer research in computer vision. Code is available at https://github.com/whai362/PVT.

Subfields: Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions (2021, background): “This is a worth exploring direction, we recently propose a solu-tion PVTv2 [67].”
Transformers in Vision: A Survey (2021, methodology): “Numerous methods have been proposed that make special design choices to perform self-attention more ‘eiciently’, for instance employing pooling/downsampling in self-attention [61, 209, 221], local window-based attention [138, 201], axial-attention [55, 83], low-rank projection attention [192, 208, 2”
Large Selective Kernel Network for Remote Sensing Object Detection (2023, methodology): “Fig 4 illustrates an LSKNet Block, which is inspired by ConvNeXt [41], MetaFormer [81], PVT-v2 [63], Conv2Former [28] and VAN [19].”
YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information (2024, methodology): “…of parameters utilization. in the field of deep learning have mainly focused on how to develop more powerful system architectures and learning methods, such as CNNs [21–23, 42, 55, 71, 72], Trans-formers [8, 9, 40, 41, 60, 69, 70], Perceivers [26, 26, 32, 52, 56, 81, 81], and Mambas [17, 38, 80].”
DaViT: Dual Attention Vision Transformers (2022, methodology): “To further reduce the computational cost, local attention [37,75,52] that limits attention in a spatially local window, and squeezed projection [58,55] that performs attention on downsampled tokens, were proposed.”
BiFormer: Vision Transformer with Bi-Level Routing Attention (2023, methodology): “These modiﬁcations include: (1) re-placing non-overlapped patch embedding [27] with overlapped one [13,35,43], (2) using deeper layout ( i.e . stacking more blocks in each stage, while reducing the base channels from 96 to 64 and MLP expansion ratio from 4 to 3 to keep similar FLOPs.)”
VMamba: Visual State Space Model (2024, background): “Moreover, other studies propose to harness the advantages of CNNs, such as introducing convolution operations [55, 7, 51], designing hybrid architectures by combining CNN and ViT modules [7, 45, 35], etc .”
Focal Modulation Networks (2022, neutral): “PVTv2-B5 [83] Panoptic SegFormer [47] 101M 54.”
Vision Transformer Adapter for Dense Predictions (2022, methodology): “To verify this, we use ViT-Adapter-S as the basic model and study 4 different attention mechanisms, including global attention [73], window attention [73], linear SRA [75], and deformable attention [85].”
Large Separable Kernel Attention: Rethinking the Large Kernel Attention Design in CNN (2023, methodology): “We also investigate the robustness of the proposed LSKA and other baselines, like LKA in VAN, ViTs [32], [37]–[39], and ConvNeXt [33], on various distortion datasets like common corruptions, semantic shift, and out-of-distribution natural adversarial examples that remain unexplored in previous…”
EMCAD: Efficient Multi-Scale Convolutional Attention Decoding for Medical Image Segmentation (2024, methodology): “Using PVTv2-b0 (Tiny) and PVTv2-b2 (Standard) encoders [56], we develop the PVT-EMCAD-B0 and PVT-EMCAD-B2 architectures.”
Medical Image Segmentation via Cascaded Attention Decoding (2023, methodology): “In PVTv2, Wang et al. [32] improve the performance of PVT by incorporating a linear complexity attention layer, an overlapping patch embedding, and a convolutional feed-forward network.”
Polyp-PVT: Polyp Segmentation with Pyramid Vision Transformers (2021, methodology): “We use PVTv2 [72] as our baseline (Bas.) and evaluate module effectiveness by removing or replacing components from the complete Polyp-PVT and comparing the variants with the standard version.”
TransNeXt: Robust Foveal Visual Perception for Vision Transformers (2023, positive): “To ensure reproducibility and consistency with prior work, we adopt the training strategy of PVTv2 [26], which incorporates various data augmentation techniques, including Random Augmentation [4], Mixup [31], CutMix [30], and Random Erasing [32].”