3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation

By Angela Dai, Matthias Nießner. European Conference on Computer Vision, 2018

356 citations recorded by the source corpus.

We present 3DMV, a novel method for 3D semantic scene segmentation of RGB-D scans in indoor environments using a joint 3D-multi-view prediction network. In contrast to existing methods that either use geometry or RGB data as input for this task, we combine both data modalities in a joint, end-to-end network architecture. Rather than simply projecting color data into a volumetric grid and operating solely in 3D – which would result in insufficient detail – we first extract feature maps from associated RGB images. These features are then mapped into the volumetric feature grid of a 3D network using a differentiable back-projection layer. Since our target is 3D scanning scenarios with possibly many frames, we use a multi-view pooling approach in order to handle a varying number of RGB input views. This learned combination of RGB and geometric features with our joint 2D-3D architecture achieves significantly better results than existing baselines. For instance, our final result on the ScanNet 3D segmentation benchmark increases from 52.8% to 75% accuracy compared to existing volumetric architectures.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Geology

View Paper: https://www.semanticscholar.org/paper/fafe7c7aa0a19d40bbbf08b9e87d650438b01b67

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Geology

Citations in the corpus

3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans (2018, methodology): “With the availability of 3D shape databases [36, 3, 32] and annotated RGB-D datasets [29, 1, 5, 2], these network architectures are now being used for 3D object classification [36, 20, 24, 28], semantic segmentation [5, 34, 6], and object or scene completion [8, 32, 9].”
Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review (2020, unknown): “B. 3D Semantic Segmentation 1) Feature-level fusion: Dai et al.[109] presented 3DMV, a multi-view network for 3D semantic segmentation which fuse image semantic and point features in voxelized point cloud.”
IPOD: Intensive Point-based Object Detector for Point Cloud (2018, unknown): “In [8, 7], a multi-view based function produces the segmentation mask.”
Atlas: End-to-End 3D Scene Reconstruction from Posed Images (2020, unknown): “3 3DMV [13] 48.”
STD: Sparse-to-Dense 3D Object Detector for Point Cloud (2019, background): “In [6, 5], a multi-view-based function produces the segmentation mask.”
Language-Grounded Indoor 3D Semantic Segmentation in the Wild (2022, background): “Early works tackled 3D semantic segmentation on dense volumetric grids [10,11], but were limited in cubic growth in memory and compute.”
Revisiting Point Cloud Shape Classification with a Simple and Effective Baseline (2021, background): “…point-cloud analysis problems like segmentation (Ladick ` y et al., 2010; Tighe & Lazeb-nik, 2010; Riemenschneider et al., 2014; Qin et al., 2018; Dai & Nießner, 2018; Kalogerakis et al., 2017; Tatarchenko et al., 2018), reconstruction (Pittaluga et al., 2019) and rendering (Aliev et al.,…”
4D Spatio-Temporal ConvNets: Minkowski Convolutional Neural Networks (2019, methodology): “9% mIoU on the ScanNet benchmark outperforming all algorithms including the best peer-reviewed work [6] by 19% mIoU at the time of submission.”
Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions (2020, background): “Current 3D scene understanding works include perception tasks such as instance segmentation [12, 21, 44, 50], semantic segmentation [34, 36, 5, 38] as well as 3D object detection and classification [40, 34, 35, 54].”
ScanQA: 3D Question Answering for Spatial Scene Understanding (2021, methodology): “Following the previous 3D and language work [10, 13], we also use additional point features such as the height of the point, colors, normals, and multiview image features [16] that project 2D appearance features to the point cloud.”
ScanRefer: 3D Object Localization in RGB-D Scans using Natural Language (2019, methodology): “Since descriptions of objects refer to attributes such as color and texture, we incorporate visual appearance by adapting the feature projection scheme in Dai and Nießner [7] to project multi-view image features ci ∈ R128 to the point cloud.”
MVTN: Multi-View Transformation Network for 3D Shape Recognition (2020, background): “pressive performance, and in many instances, achieve stateof-the-art results in 3D shape classification and segmentation [40, 75, 45, 39, 16].”
Point Transformer V3: Simpler, Faster, Stronger (2023, unknown): “However, in contrast to the progress made in 2D vision or NLP, the development of 3D backbones [16, 46, 62, 88] has been hindered in terms of scale, primarily due to the limited size and diversity of point cloud data available in separate domains [92].”
Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review (2020, methodology): “3DMV [77] combines the geometry and imagery data as input to train a joint 3D deep architecture.”
Monte Carlo convolution for learning on non-uniformly sampled point clouds (2018, methodology): “We report performance as mean per-class voxel accuracy [Dai and Nießner 2018]; a more meaningful measure than the original ScanNet metric (overall voxel accuracy) used on PointNet++.”
Geometric Deep Learning for Volumetric Computational Fluid Dynamics (2019, methodology): “A similar approach of feature extraction from a lower dimension data has been used for semantic scene segmentation in [55] where rather than projecting directly RGB images into the 3D grid, features were extracted and these later were projected to improve the accuracy.”