Feature Pyramid Transformer
By Dong Zhang, Hanwang Zhang, Jinhui Tang, Meng Wang, Xiansheng Hua, Qianru Sun. 2020
306 citations recorded by the source corpus.
Feature interactions across space and scales underpin modern visual recognition systems because they introduce beneficial visual contexts. Conventionally, spatial contexts are passively hidden in the CNN's increasing receptive fields or actively encoded by non-local convolution. Yet, the non-local spatial interactions are not across scales, and thus they fail to capture the non-local contexts of objects (or parts) residing in different scales. To this end, we propose a fully active feature interaction across both space and scales, called Feature Pyramid Transformer (FPT). It transforms any feature pyramid into another feature pyramid of the same size but with richer contexts, by using three specially designed transformers in self-level, top-down, and bottom-up interaction fashion. FPT serves as a generic visual backbone with fair computational overhead. We conduct extensive experiments in both instance-level (i.e., object detection and instance segmentation) and pixel-level segmentation tasks, using various backbones and head networks, and observe consistent improvement over all the baselines and the state-of-the-art methods.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
Citations in the corpus
- Meta-attention for ViT-backed Continual Learning (2022, background): “The transformer [46], a prevailing network architecture in nature language processing (NLP) [1, 9, 35], has received growing interest and achieved great accomplishment in the computer vision field, enjoying state-of-the-art performance on many visual tasks, including image classification [10, 28, 45”
- Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “Recently, transformer architectures [7, 26, 86, 97, 104, 109] shine in semantic segmentation, focusing on multi-scale feature fusion [12, 84, 89, 101] and contextual feature aggregation [58, 79].”
- MOTR: End-to-End Multiple-Object Tracking with TRansformer (2021, methodology): “FPT [43] employs feature pyramid transformer architecture to fuse and interact multiscale features for object detection.”
- Remote Sensing Image Change Detection With Transformers (2021, background): “models show comparable or even better performance as the convolutional counterparts in various visual tasks, including image classification [33]–[35], segmentation [35]–[37], object detection [36], [38], [39], image generation [40], [41], image captioning [42], and super-resolution [43], [44].”
- Transformer-Based Visual Segmentation: A Survey (2023, methodology): “The subfields include point cloud segmentation, domain-aware segmentation, label and model efficient segmentation, class agnostic segmentation, tracking, and medical segmentation.”
- SwinSUNet: Pure Transformer Network for Remote Sensing Image Change Detection (2022, methodology): “Due to the amazing semantic representation capability of the Transformer, it performs as well as or even better than CNN in various CV tasks, such as image classification [29-32], object detection [33-35], semantic segmentation [31,33,36], image generation [37,38], super-resolution [39,40], etc.”
- A Survey of Visual Transformers (2021, methodology): “Specialized: FPT [94], HRFormer [95], HRViT [96].”
- AFPN: Asymptotic Feature Pyramid Network for Object Detection (2023, unknown): “For example, FPT [12] introduces the self-attention mechanism in the NLP field to extract features at different levels and uses a multi-scale attention network to aggregate these features.”
- A Survey on Vision Transformer (2020, methodology): “[145] suggest that traditional methods failed to interact the cross-scale features, and thus propose Feature Pyramid Transformer (FPT) to fully exploit feature interactions across both space and scales.”