Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation

By Xiaokang Chen, Kwan-Yee Lin, Jingbo Wang, Wayne Wu, Chen Qian, Hongsheng Li, Gang Zeng. 2020

449 citations recorded by the source corpus.

Depth information has proven to be a useful cue in the semantic segmentation of RGB-D images for providing a geometric counterpart to the RGB representation. Most existing works simply assume that depth measurements are accurate and well-aligned with the RGB pixels and models the problem as a cross-modal feature fusion to obtain better feature representations to achieve more accurate segmentation. This, however, may not lead to satisfactory results as actual depth data are generally noisy, which might worsen the accuracy as the networks go deeper. In this paper, we propose a unified and efficient Cross-modality Guided Encoder to not only effectively recalibrate RGB feature responses, but also to distill accurate depth information via multiple stages and aggregate the two recalibrated representations alternatively. The key of the proposed architecture is a novel Separation-and-Aggregation Gating operation that jointly filters and recalibrates both representations before cross-modality aggregation. Meanwhile, a Bi-direction Multi-step Propagation strategy is introduced, on the one hand, to help to propagate and fuse information between the two modalities, and on the other hand, to preserve their specificity along the long-term propagation process. Besides, our proposed encoder can be easily injected into the previous encoder-decoder structures to boost their performance on RGB-D semantic segmentation. Our model outperforms state-of-the-arts consistently on both in-door and out-door challenging datasets. Code of this work is available at this https URL

Subfields: Computer Vision and Pattern Recognition, Industrial and Manufacturing Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Cross-level Feature Aggregation Network for Polyp Segmentation (2023, methodology): “Considering that noise could exist in the low-level features, a gated fusion strategy [45] is adopted to integrate the two features.”
A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation (2024, methodology): “…benchmarked the performance of the proposed FTransUNet against twelve representative state-of-the-art methods, namely ABCNet [61], PSPNet [62], MAResU-Net [63], vFuseNet [15], FuseNet [14], ESANet [33], Tran-sUNet [31], SA-GATE [64], UNetFormer [44], CMFNet [19], MFTransNet [59], and CMGFNet [47].”
Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis (2020, methodology): “The majority of approaches for RGB-D segmentation [9], [21], [10], [11], [22], [23], [24], [25] simply use two branches, one for RGB and one for depth data and fuse the feature representations later in the network.”
Changer: Feature Interaction is What You Need for Change Detection (2022, methodology): “We abstract the aggregation-distribution style feature interaction from co-attention and some similar mechanisms [8,20,26,28], as shown in Fig.”