3D-SIS: 3D Semantic Instance Segmentation of RGB-D Scans

By Ji Hou, Angela Dai, Matthias Nießner. 2018

548 citations recorded by the source corpus.

We introduce 3D-SIS, a novel neural network architecture for 3D semantic instance segmentation in commodity RGB-D scans. The core idea of our method to jointly learn from both geometric and color signal, thus enabling accurate instance predictions. Rather than operate solely on 2D frames, we observe that most computer vision applications have multi-view RGB-D input available, which we leverage to construct an approach for 3D instance segmentation that effectively fuses together these multi-modal inputs. Our network leverages high-resolution RGB input by associating 2D images with the volumetric grid based on the pose alignment of the 3D reconstruction. For each image, we first extract 2D features for each pixel with a series of 2D convolutions; we then backproject the resulting feature vector to the associated voxel in the 3D grid. This combination of 2D and 3D feature learning allows significantly higher accuracy object detection and instance segmentation than state-of-the-art alternatives. We show results on both synthetic and real-world public benchmarks, achieving an improvement in mAP of over 13 on real-world data.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Geology, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Geology

In 3D Vision & Pose

Citations in the corpus

3D Instance Segmentation of MVS Buildings (2021, neutral): “methods [5], [6] are limited by memory and computing power, thus are mainly used for the segmentation of relatively small indoor scenes [7], [8], [9], [10], [11], [12], which are not suitable for large-scale outdoor scenes.”
From Points to Parts: 3D Object Detection From Point Cloud With Part-Aware and Part-Aggregation Network (2019, background): “[47] combined the multi-view RGB images and 3D point cloud to better generate proposals and predict object instance masks in and end-to-end manner.”
Mask3D: Mask Transformer for 3D Semantic Instance Segmentation (2022, methodology): “While 3D-SIS [22] relies on predefined anchor boxes [19], 3D-BoNet [61] proposes an interesting variation that predicts bounding boxes from a global scene descriptor and optimizes an association”
Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, background): “3D-SIS [Hou et al., 2019] leverages joint 2D and 3D end-to-end feature learning on both geometry and RGB input for 3D object bounding box detection and semantic instance segmentation.”
Deep Learning for Image and Point Cloud Fusion in Autonomous Driving: A Review (2020, methodology): “presented 3D-SIS [114], a two-stage 3D CNN that performs voxel-wise 3D instance segmentation on multi-view images and RGB-D scan data.”
MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation (2019, methodology): “Inspired by the recent success of Mask R-CNN [5] on 2D instance segmentation, 3D-SIS [6] develops a proposal-based system which achieves the state-of-the-art performance on 3D instance segmentation evaluated using the ScanNet benchmark [2].”
Learning 3D Semantic Scene Graphs From 3D Indoor Reconstructions (2020, background): “Current 3D scene understanding works include perception tasks such as instance segmentation [12, 21, 44, 50], semantic segmentation [34, 36, 5, 38] as well as 3D object detection and classification [40, 34, 35, 54].”
3D Object Detection with Pointformer (2020, background): “proaches [28, 12, 42] gridify the irregular point clouds into regular voxels and are followed by sparse 3D convolutions to learn high dimensional features.”
OccuSeg: Occupancy-Aware 3D Instance Segmentation (2020, methodology): “Pioneer works [43, 40, 17] have tried to directly extend 2D convolutional neural networks to 3D space by voxelizing the input points into uniform voxels, and applying 3D convolutions instead.”
Deep Hough Voting for 3D Object Detection in Point Clouds (2019, methodology): “Deep sliding shapes (DSS) [42] and 3D-SIS [12] are both 3D CNN based detectors that combine geometry and RGB cues in object proposal and classification, based on the Faster R-CNN [37] pipeline.”
Atlas: End-to-End 3D Scene Reconstruction from Posed Images (2020, methodology): “3D-SIS [25] improves upon these approaches by fusing 2D features in a 3D representation.”
PointContrast: Unsupervised Pre-training for 3D Point Cloud Understanding (2020, background): “Among many different choices [73, 28, 44, 42], we identify the recently proposed VoteNet [43] as a good candidate for three main reasons.”
Back to Reality: Weakly-supervised 3D Object Detection with Shape-guided Label Enhancement (2022, background): “More recently, networks that directly consume point clouds have been proposed [11, 20, 25, 35, 51].”
Unsupervised 3D Shape Completion through GAN Inversion (2021, background): “3D shape completion has played an important role for robotics [11, 24] and perception [9, 14].”
ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data (2021, background): “A variety of methods are addressing different parts of the challenge, like depth estimation, 3D reconstruction, instance segmentation [1, 2, 3], object detection [4, 5, 6, 7] and more.”
Exploiting Depth Information for Wildlife Monitoring (2021, background): “Other approaches are highly specialized, e.g. by requiring multiple perspectives on a scene (Hou et al., 2018) or being highly reliant on the spatial set-up of the scene observation (Shao et al., 2018).”
ImVoteNet: Boosting 3D Object Detection in Point Clouds With Image Votes (2020, background): “Another line of work [45, 18, 48, 11] takes a more 3D-focused way to concatenate intermediate ConvNet features from 2D images to 3D voxels or points to enrich 3D features, before they are used for object proposal and box regression.”
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, background): “3D instance segmentation goes a step further by distinguishing multiple objects belonging to the same semantic category, predicting individual masks for each object instance [11, 14, 22, 24, 31, 37, 55, 60, 62, 67].”