RDFNet: RGB-D Multi-level Residual Feature Fusion for Indoor Semantic Segmentation

By Seungyong Lee, Seong-Jin Park, Ki Sang Hong. IEEE International Conference on Computer Vision, 2017

335 citations recorded by the source corpus.

In multi-class indoor semantic segmentation using RGB-D data, it has been shown that incorporating depth feature into RGB feature is helpful to improve segmentation accuracy. However, previous studies have not fully exploited the potentials of multi-modal feature fusion, e.g., simply concatenating RGB and depth features or averaging RGB and depth score maps. To learn the optimal fusion of multimodal features, this paper presents a novel network that extends the core idea of residual learning to RGB-D semantic segmentation. Our network effectively captures multilevel RGB-D CNN features by including multi-modal feature fusion blocks and multi-level feature refinement blocks. Feature fusion blocks learn residual RGB and depth features and their combinations to fully exploit the complementary characteristics of RGB and depth data. Feature refinement blocks learn the combination of fused features from multiple levels to enable high-resolution prediction. Our network can efficiently train discriminative multi-level features from each modality end-to-end by taking full advantage of skip-connections. Our comprehensive experiments demonstrate that the proposed architecture achieves the state-of-the-art accuracy on two challenging RGB-D indoor datasets, NYUDv2 and SUN RGB-D.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e637eb1ddf31f8d43287b881d622fff4174758a5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Ming-Ming Cheng
3. Ming-Hsuan Yang
4. Lei Zhang
5. Ling Shao
6. Hongsheng Li
7. Yi Yang
8. Xiaogang Wang

Citations in the corpus

• Bi-directional Cross-Modality Feature Propagation with Separation-and-Aggregation Gate for RGB-D Semantic Segmentation (2020, positive): “Standard practice is to use the depth data 2 as another input and adopt Fully Convolutional Network (FCN)-like architectures with feature fusion schemas, e.g., convolution and modality-based affinity etc., to fuse the features of two modalities [26,6,17,38].”
• Joint Task-Recursive Learning for Semantic Segmentation and Depth Estimation (2018, methodology): “[53], our TRL with ResNet-50 has a total 2.”
• Deep Multimodal Fusion by Channel Exchanging (2020, methodology): “Joining the success of deep learning, multimodal fusion is recently specified as deep multimodal fusion by introducing end-to-end neural integration of multiple modalities [37], and it has exhibited remarkable benefits against the unimodal paradigm in semantic segmentation [28, 44], action recogniti”
• Beyond RGB: Very High Resolution Urban Remote Sensing With Multimodal Deep Networks (2017, background): “Finally, the recent work of [23] builds on the FuseNet architecture to incorporate residual learning and multiple stages of refinement to obtain high resolution multi-modal predictions RGB-D data.”
• Depth-aware CNN for RGB-D Segmentation (2018, background): “ the RGB settings, RGB-D segmentation is able to integrate geometry into scene understanding. In [14,15,16,17], depth is simply treated as additional channels and directly fed into CNN. Several works [1,16,18,19,20] encode depth to HHA image, which has three channels: horizontal disparity, height ab”
• Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation (2020, methodology): “1 RDFNet [5]ICCV 17 2×ResNet101 3 HHA 75.”
• Progressively Complementarity-Aware Fusion Network for RGB-D Salient Object Detection (2018, methodology): “To our best knowledge, previous works mainly adopt residual connections to reuse features of preceding layers flexibly inside unimodal network streams [21, 28-30], while in this work, the residual connection is introduced in a cross-modal way in multiple levels for fusing RGB-D image pairs.”
• Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis (2020, methodology): “The majority of approaches for RGB-D segmentation [9], [21], [10], [11], [22], [23], [24], [25] simply use two branches, one for RGB and one for depth data and fuse the feature representations later in the network.”
• What Makes Multimodal Learning Better than Single (Provably) (2021, background): “For example, [35, 23, 22, 31, 20] combine RGB and depth images to improve semantic segmentation; [8, 16] fuse audio with video to do scene understanding; researchers also explore audio-visual source separation and localization [49, 11].”
• Monocular depth estimation based on deep learning: An overview (2020, methodology): “be widely used in simultaneous localization and mapping (SLAM) [1], navigation [2], object detection [3] and semantic segmentation [4], etc.”
• Survey on semantic segmentation using deep learning techniques (2019, methodology): “Inspired by RefineNet, Park et al. [89] proposed RGB-D fusion network (RDFNet) for semantic segmentation.”