Spatial Information Guided Convolution for Real-Time RGBD Semantic Segmentation

By Lin-Zhuo Chen, Zheng Lin, Ziqin Wang, Yong-Liang Yang, Ming-Ming Cheng. 2020

164 citations recorded by the source corpus.

3D spatial information is known to be beneficial to the semantic segmentation task. Most existing methods take 3D spatial data as an additional input, leading to a two-stream segmentation network that processes RGB and 3D spatial information separately. This solution greatly increases the inference time and severely limits its scope for real-time applications. To solve this problem, we propose Spatial information guided Convolution (S-Conv), which allows efficient RGB feature and 3D spatial information integration. S-Conv is competent to infer the sampling offset of the convolution kernel guided by the 3D spatial information, helping the convolutional layer adjust the receptive field and adapt to geometric transformations. S-Conv also incorporates geometric information into the feature learning process by generating spatially adaptive convolutional weights. The capability of perceiving geometry is largely enhanced without much affecting the amount of parameters and computational cost. Based on S-Conv, we further design a semantic segmentation network, called Spatial information Guided convolutional Network (SGNet), resulting in real-time inference and state-of-the-art performance on NYUDv2 and SUNRGBD datasets.

Subfields: Aerospace Engineering, Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computational Mechanics

Horst-Michael Groß

In Computer Vision and Pattern Recognition

Citations in the corpus

Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “However, these fully-supervised models [16, 23, 93] can only handle a fixed number of classes.”
Efficient RGB-D Semantic Segmentation for Indoor Scene Analysis (2020, background): “[16], [17], [18], [19], [20] design specifically tailored convolutions, taking into account depth information.”