MASC: Multi-scale Affinity with Sparse Convolution for 3D Instance Segmentation

By Chen Liu, Yasutaka Furukawa. 2019

87 citations recorded by the source corpus.

We propose a new approach for 3D instance segmentation based on sparse convolution and point affinity prediction, which indicates the likelihood of two points belonging to the same instance. The proposed network, built upon submanifold sparse convolution [3], processes a voxelized point cloud and predicts semantic scores for each occupied voxel as well as the affinity between neighboring voxels at different scales. A simple yet effective clustering algorithm segments points into instances based on the predicted affinity and the mesh topology. The semantic for each instance is determined by the semantic prediction. Experiments show that our method outperforms the state-of-the-art instance segmentation methods by a large margin on the widely used ScanNet benchmark [2]. We share our code publicly at this https URL.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Geology, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Li Jiang
2. Or Litany
3. Shaoshuai Shi
4. Jiaya Jia
5. Hengshuang Zhao
6. Chi-Wing Fu
7. Bastian Leibe
8. Yulan Guo

In Computer Graphics and Computer-Aided Design

1. Sen Wang
2. R. Clark
3. Shiguo Lian
4. Chi-Wing Fu
5. Lei Han
6. Lu Fang
7. Mohammed Bennamoun
8. Jiaya Jia

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Xiaogang Wang
3. Ming-Hsuan Yang
4. Dacheng Tao
5. Andrea Vedaldi
6. Jiaya Jia
7. Bernt Schiele
8. Yi Yang

In Geology

1. Or Litany
2. Li Jiang
3. Shaoshuai Shi
4. Jiaya Jia
5. Hengshuang Zhao
6. Chi-Wing Fu
7. Qingyong Hu
8. Shu Liu

In 3D Vision & Pose

1. Yulan Guo
2. Sen Wang
3. R. Clark
4. Shiguo Lian
5. Chi-Wing Fu
6. Lei Han
7. Lu Fang
8. Mohammed Bennamoun

Citations in the corpus

• 3D Instance Segmentation of MVS Buildings (2021, methodology): “Volumetric methods ease this issue using a 3D grid representation, which is notoriously expensive in terms of computational efficiency and memory consumption [4], [12], [36], [37], [38], [39].”
• OccuSeg: Occupancy-Aware 3D Instance Segmentation (2020, methodology): “While volumetric approaches [26, 25, 21] employ Spatially-Sparse Convolutional Networks (SSCN) [12, 2] to omit computations on inactive voxels using the sparse convolution technique.”
• Vision-based robotic grasping from object localization, object pose estimation to grasp estimation for parallel grippers: a review (2020, methodology): “MASC [Liu and Furukawa, 2019] utilizes the submanifold sparse convolutions [Graham and van der Maaten, 2017; Graham et al.”
• Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds (2019, methodology): “ASIS [51], JSIS3D [34], MASC [30], 3D-BEVIS [8] and [28] use the same strategy to group point-level features for instance segmentation.”
• Deep Learning for 3D Point Clouds: A Survey (2019, methodology): “Similarly, Liu et al. [201] first leveraged submanifold sparse convolution [155] to predict semantic scores of each voxel and affinity between neighboring voxels.”
• Mask3D: Mask Transformer for 3D Semantic Instance Segmentation (2022, methodology): “As a result, bottomup approaches which benefit from strong features (MTML [28], MASC [32]) experienced another performance boost.”
• PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation (2020, background): “Liu and Furukawa [27] predict both the semantic labels and affinity between adjacent voxels in different scales to group instances.”