Depth-aware CNN for RGB-D Segmentation

By Weiyue Wang, Ulrich Neumann. 2018

292 citations recorded by the source corpus.

Convolutional neural networks (CNN) are limited by the lack of capability to handle geometric information due to the fixed grid kernel structure. The availability of depth data enables progress in RGB-D semantic segmentation with CNNs. State-of-the-art methods either use depth as additional images or process spatial information in 3D volumes or point clouds. These methods suffer from high computation and memory cost. To address these issues, we present Depth-aware CNN by introducing two intuitive, flexible and effective operations: depth-aware convolution and depth-aware average pooling. By leveraging depth similarity between pixels in the process of information propagation, geometry is seamlessly incorporated into CNN. Without introducing any additional parameters, both operators can be easily integrated into existing CNNs. Extensive experiments and ablation studies on challenging RGB-D semantic segmentation benchmarks validate the effectiveness and flexibility of our approach.

Subfields: Computer Vision and Pattern Recognition, Geology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, background): “Later work [11, 19, 20] interpret this “modulation” as a way to make convolution spatially adaptive and content-aware.”
Image Segmentation Using Deep Learning: A Survey (2020, neutral): “9 D-aware CNN [179] 61.”
Deep Learning for Image Super-Resolution: A Survey (2019, background): “, distance) between the viewpoint and objects in the scene, and plays important roles in many tasks like pose estimation [161], [162] and semantic segmentation [163], [164].”
BinsFormer: Revisiting Adaptive Bins for Monocular Depth Estimation (2022, background): “A widespread of various depth-depended tasks related to autonomous driving [2], [3], [4], [5], [6], virtual reality [7], [8], and scene understanding [9], [10], [11], [12] provide strong demand for effective MDE methods that can accurately reconstruct the 3D world.”
SNE-RoadSeg: Incorporating Surface Normal Information into Semantic Segmentation for Accurate Freespace Detection (2020, background): “Although FuseNet [19] demonstrates impressive performance, the ability of CNNs to handle geometric information is limited, due to the fixed grid kernel structure [35].”
Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis (2020, background): “[16], [17], [18], [19], [20] design specifically tailored convolutions, taking into account depth information.”
Self-Supervised Drivable Area and Road Anomaly Segmentation Using RGB-D Data For Robotic Wheelchairs (2019, methodology): “segmentation neural networks that fuse RGB and depth data together such as FuseNet [5] and Depth-aware CNN [6], which can achieve impressive performance.”
Dynamic Fusion Module Evolves Drivable Area and Road Anomaly Detection: A Benchmark and Algorithms (2021, methodology): “With the rapid development of deep-learning technologies, many semantic segmentation approaches based on convolutional neural networks (CNNs) have been proposed, and these methods can be used for drivable area and road anomaly detection.”
ACNET: Attention Based Network to Exploit Complementary Features for RGBD Semantic Segmentation (2019, background): “In [7], traditional convolution was modified according to depth values.”
Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation (2020, methodology): “Here we compare our S-Conv with two-stream network, deformable convolution [13], [14], and depth-aware convolution [16].”
Deep Surface Normal Estimation With Hierarchical RGB-D Fusion (2019, methodology): “, semantic segmentation [27, 5, 23], object detection [11, 20], 3D reconstruction [15, 18, 12], etc.”
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers (2022, methodology): “Following the data splitting [15], [66], areas of {1, 2, 3, 4, 6} are used for training and area 5 is for testing.”
Unseen Object Instance Segmentation for Robotic Environments (2020, methodology): “[16] also used the HHA in their pioneering work on FCNs. [32] proposed a depth-aware convolution and pooling mechanism to incorporate geometry into the convolution operators to build a depth-aware receptive ﬁeld.”