ACNET: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation

By Xinxin Hu, Kailun Yang, Lei Fei, Kaiwei Wang. International Conference on Information Photonics, 2019

456 citations recorded by the source corpus.

Compared to RGB semantic segmentation, RGBD semantic segmentation can achieve better performance by taking depth information into consideration. However, it is still problematic for contemporary segmenters to effectively exploit RGBD information since the feature distributions of RGB and depth (D) images vary significantly in different scenes. In this paper, we propose an Attention Complementary Network (ACNet) that selectively gathers features from RGB and depth branches. The main contributions lie in the Attention Complementary Module (ACM) and the architecture with three parallel branches. More precisely, ACM is a channel attention-based module that extracts weighted features from RGB and depth branches. The architecture preserves the inference of the original RGB and depth branches, and enables the fusion branch at the same time. Based on the above structures, ACNet is capable of exploiting more high-quality features from different channels. We evaluate our model on SUN-RGBD and NYUDv2 datasets, and prove that our model outperforms state-of-the-art methods. In particular, a mIoU score of 48.3% on NYUDv2 test set is achieved with ResNet50. We will release our source code based on PyTorch and the trained segmentation model at https://github.com/anheidelonghu/ACNet.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a711b3bb3059d0b9b56f38897007c50ad059bf96

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

BBS-Net: RGB-D Salient Object Detection with a Bifurcated Backbone Strategy Network (2020, background): “. yet eective progressively transposed module (PTM, Fig.3(d)) to predict the nal saliency map (S 2) in a progressive upsampling way. It is composed of two sequential residual-based transposed blocks [33] and three sequential 1 1 convolutions. Each residual-based transposed block consists of a 3 3 co”
Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis (2020, methodology): “It further prevents gridding artifacts introduced by transposed convolutions as used in ACNet [11] or RedNet [10].”
Deep Multimodal Data Fusion (2024, unknown): “Unlike the above works that use RGB branch as the backbone of feature extraction, Hu et al. [64] build a new fusion sub-network as the backbone.”
Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation (2020, methodology): “3) Comparison With Alternatives: Most methods [6], [9], [33], [62] use a two-stream network to extract features from two different modalities and then combine them.”
Deep Multimodal Fusion by Channel Exchanging (2020, unknown): “3 ACNet [22] ResNet50 - - 48.”
Is Context-Aware CNN Ready for the Surroundings? Panoramic Semantic Segmentation in the Wild (2021, methodology): “Unlike the squeeze&excitation components that use fully connected layers to boost the informative features in classiﬁcation and pinhole image segmentation tasks [20], [21], [24], we leverage convolution layers.”
Identification of Plant-Leaf Diseases Using CNN and Transfer-Learning Approach (2021, methodology): “In addition, we aim to carry this research forward by implementing it with a new deep-learning model, such as ACNet [57], and a transformer-based architecture, such as ViT [58] and the MLP Mixer [59] method, in plant disease identiﬁcation, and evaluate its performance.”
What Makes Multimodal Learning Better than Single (Provably) (2021, background): “For example, [35, 23, 22, 31, 20] combine RGB and depth images to improve semantic segmentation; [8, 16] fuse audio with video to do scene understanding; researchers also explore audio-visual source separation and localization [49, 11].”
Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation (2020, unknown): “Standard practice is to use the depth data 2 as another input and adopt Fully Convolutional Network (FCN)-like architectures with feature fusion schemas, e.g., convolution and modality-based aﬃnity etc., to fuse the features of two modalities [26,6,17,38].”