FaPN: Feature-aligned Pyramid Network for Dense Image Prediction

By Shihua Huang, Zhichao Lu, Ran Cheng, Cheng He. 2021

262 citations recorded by the source corpus.

Recent advancements in deep neural networks have made remarkable leap-forwards in dense image prediction. However, the issue of feature alignment remains as neglected by most existing approaches for simplicity. Direct pixel addition between upsampled and local features leads to feature maps with misaligned contexts that, in turn, translate to mis-classifications in prediction, especially on object boundaries. In this paper, we propose a feature alignment module that learns transformation offsets of pixels to contextually align upsampled higher-level features; and another feature selection module to emphasize the lower-level features with rich spatial details. We then integrate these two modules in a top-down pyramidal architecture and present the Feature-aligned Pyramid Network (FaPN). Extensive experimental evaluations on four dense prediction tasks and four datasets have demonstrated the efficacy of FaPN, yielding an overall improvement of 1.2 - 2.6 points in AP / mIoU over FPN when paired with Faster / Mask R-CNN. In particular, our FaPN achieves the state-of-the-art of 56.7% mIoU on ADE20K when integrated within Mask-Former. The code is available from https://github.com/EMI-Group/FaPN.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “However, recent work still focuses on advancing specialized architectures [20, 39, 45], which raises the question: why haven’t universal architectures replaced specialized ones?”
Accurate Leukocyte Detection Based on Deformable-DETR and Multi-Level Feature Fusion for Aiding Diagnosis of Blood Diseases (2024, methodology): “To demonstrate the effectiveness of the HS-FPN module in fusing multi-scale features, we compared it with other multi-scale feature fusion methods, such as FPN ([13]), BiFPN ([25]), PaFPN ([16]), and FaPN ([9]).”
Changer: Feature Interaction is What You Need for Change Detection (2022, methodology): “Moreover, alignment-based fusion methods focus on feature alignment, and they typically use ﬂow ﬁeld or de-formable convolution [10, 50] to align features of different levels in the spatial dimension [16,17,21,34,38].”
Vision Transformer Adapter for Dense Predictions (2022, unknown): “3 Swin-L-FaPN [33] Mask2Former [12] 640×640 56.”
Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization (2022, unknown): “, [5, 15, 24]), few had pretrained semantic segmentation models for ADE20k [72] and were compatible with our inference toolchain (ONNX and TensorRT).”