P2T: Pyramid Pooling Transformer for Scene Understanding

By Yu-Huan Wu, Yun Liu, Mingg-Ming Cheng. 2021

313 citations recorded by the source corpus.

Recently, the vision transformer has achieved great success by pushing the state-of-the-art of various vision tasks. One of the most challenging problems in the vision transformer is that the large sequence length of image tokens leads to high computational cost (quadratic complexity). A popular solution to this problem is to use a single pooling operation to reduce the sequence length. This paper considers how to improve existing vision transformers, where the pooled feature extracted by a single pooling operation seems less powerful. To this end, we note that pyramid pooling has been demonstrated to be effective in various vision tasks owing to its powerful ability in context abstraction. However, pyramid pooling has not been explored in backbone network design. To bridge this gap, we propose to adapt pyramid pooling to Multi-Head Self-Attention (MHSA) in the vision transformer, simultaneously reducing the sequence length and capturing powerful contextual features. Plugged with our pooling-based MHSA, we build a universal vision transformer backbone, dubbed Pyramid Pooling Transformer (P2T). Extensive experiments demonstrate that, when applied P2T as the backbone network, it shows substantial superiority in various vision tasks such as image classification, semantic segmentation, object detection, and instance segmentation, compared to previous CNN- and transformer-based networks. The code will be released at https://github.com/yuhuan-wu/P2T.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Transformers in Vision: A Survey (2021, methodology): “Numerous methods have been proposed that make special design choices to perform self-attention more “efficiently,” for instance employing pooling/downsampling in self-attention [61, 208, 220], local window-based attention [137, 200],”
Localization Distillation for Object Detection (2022, methodology): “Cross architecture distillation using recent state-of-the-art classification models [11], [20], [25], [43], [67] as teachers is also an interesting direction to explore.”
JCS: An Explainable COVID-19 Diagnosis System by Joint Classification and Segmentation (2020, background): “Some of the representatives [85]–[88] can largely surpass CNN networks with varies of vision tasks such as image classification, object detection, and semantic segmentation.”
Visual attention network (2022, background): “, vision transformers) [13], [15], [20], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], [64] have achieved significantly better performance than the mainstream CNNs on different visual tasks, showing the huge potential of attention-based models.”
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers (2022, background): “Following this trend, various architectures of dense prediction transformers [67], [68], [69], [70], [71] and semantic segmentation transformers [72], [73], [74], [75], [76] emerge in the field.”
Large Selective Kernel Network for Remote Sensing Object Detection (2023, methodology): “Transformer-based [59] models, such as the Vision Transformer (ViT) [1,11,14,53,60], Swin transformer [25, 39, 51, 64, 70, 83], and pyramid transformer [62, 67], have gained popularity in computer vision.”
Vision Transformer Adapter for Dense Predictions (2022, methodology): “However, under regular training settings ( i.e. , apply ImageNet supervised pre-training and ﬁne-tune for 36 epochs), their detection performance is still inferior 1 to recent models (Chu et al., 2021b; Dong et al., 2021; Wang et al., 2022a; Wu et al., 2022b) that well combine image prior.”