Anisotropic Convolutional Networks for 3D Semantic Scene Completion

By J. Li, Kai Han, Peng Wang, Yu Liu, Xia Yuan. 2020

187 citations recorded by the source corpus.

As a voxel-wise labeling task, semantic scene completion (SSC) tries to simultaneously infer the occupancy and semantic labels for a scene from a single depth and/or RGB image. The key challenge for SSC is how to effectively take advantage of the 3D context to model various objects or stuffs with severe variations in shapes, layouts, and visibility. To handle such variations, we propose a novel module called anisotropic convolution, which properties with flexibility and power impossible for the competing methods such as standard 3D convolution and some of its variations. In contrast to the standard 3D convolution that is limited to a fixed 3D receptive field, our module is capable of modeling the dimensional anisotropy voxel-wisely. The basic idea is to enable anisotropic 3D receptive field by decomposing a 3D convolution into three consecutive 1D convolutions, and the kernel size for each such 1D convolution is adaptively determined on the fly. By stacking multiple such anisotropic convolution modules, the voxel-wise modeling capability can be further enhanced while maintaining a controllable amount of model parameters. Extensive experiments on two SSC benchmarks, NYU-Depth-v2 and NYUCAD, show the superior performance of the proposed method.

Subfields: Aerospace Engineering, Computational Mechanics, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Yunpeng Zhang
2. Zhengbiao Zhu
3. Gao Huang
4. Jiwen Lu
5. Shiji Song
6. Anh-Quan Cao
7. Dalong Du
8. Jie Zhou

In Computational Mechanics

1. Zhengbiao Zhu
2. Yunpeng Zhang
3. Dalong Du
4. Gao Huang
5. Jiwen Lu
6. Shiji Song
7. Anh-Quan Cao
8. Jie Zhou

In Computer Vision and Pattern Recognition

1. Shiji Song
2. Yulin Wang
3. Gao Huang
4. Jie Zhou
5. Jiwen Lu
6. Yizeng Han
7. Zhengbiao Zhu
8. Dalong Du

In 3D Vision & Pose

1. Zhengbiao Zhu
2. Yunpeng Zhang
3. Dalong Du
4. Jiwen Lu
5. Anh-Quan Cao
6. Jie Zhou
7. Raoul de Charette

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, background): “Point Cloud I & S 3D Shape classification and segmentation, 3D scene segmentation [217], 3D semantic scene completion [218]”
• MonoScene: Monocular 3D Semantic Scene Completion (2021, methodology): “Input LMSCNetrgb [55] AICNetrgb [39] 3DSketchrgb [9] MonoScene (ours) Ground Truth”
• Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving (2023, unknown): “Another related task is Semantic Scene Completion (SSC) [1, 6, 9, 3, 24, 50, 38, 49, 7, 19, 50, 32, 20, 29, 54], whose goal is to estimate a dense semantic space from partial observations.”
• Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction (2023, background): “The ability to describe fine-grained 3D structures makes voxel-based representation favorable for 3D semantic occupancy prediction tasks including lidar segmentation [12, 29, 40, 44, 45, 51] and 3D scene completion [5, 10, 23, 38, 43].”
• OccFormer: Dual-path Transformer for Vision-based 3D Semantic Occupancy Prediction (2023, methodology): “For the driving scenes, most existing methods [43, 11, 8, 25, 50] still rely on the expensive Li-DAR sensors for explicit depth measurements.”