Associatively Segmenting Instances and Semantics in Point Clouds

By Xinlong Wang, Shu Liu, Xiaoyong Shen, Chunhua Shen, Jiaya Jia. 2019

285 citations recorded by the source corpus.

A 3D point cloud describes the real scene precisely and intuitively. To date how to segment diversified elements in such an informative 3D scene is rarely discussed. In this paper, we first introduce a simple and flexible framework to segment instances and semantics in point clouds simultaneously. Then, we propose two approaches which make the two tasks take advantage of each other, leading to a win-win situation. Specifically, we make instance segmentation benefit from semantic segmentation through learning semantic-aware point-level instance embedding. Meanwhile, semantic features of the points belonging to the same instance are fused together to make more accurate per-point semantic predictions. Our method largely outperforms the state-of-the-art method in 3D instance segmentation along with a significant improvement in 3D semantic segmentation. Code has been made available at: https://github.com/WXinlong/ASIS.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Environmental Engineering, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Environmental Engineering

In 3D Vision & Pose

Citations in the corpus

3D Instance Segmentation of MVS Buildings (2021, background): “JSNet [41] and ASIS [10] both learn the instance embedding space and combine semantic features and instance features of the point clouds to jointly improve the”
From Points to Parts: 3D Object Detection From Point Cloud With Part-Aware and Part-Aggregation Network (2019, background): “[49] proposed a semantic-aware point-level instance embedding strategy to learn better features for both the semantic and instance point cloud segmentation.”
Mask3D: Mask Transformer for 3D Semantic Instance Segmentation (2022, methodology): “We report scores using the mAP metric from ScanNet and mean precision/recall at IoU threshold 50% (mPrec50/mRec50) as initially introduced by ASIS [59].”
PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation (2020, background): “7 points higher than ASIS [50], respectively.”
Self-Supervised Pretraining of 3D Features on any Point-Cloud (2021, methodology): “Point-cloud based models [66, 68] are widely used in classification and segmentation tasks [6, 40, 45, 66, 68, 91, 98, 100, 106, 107], 3D reconstruction [23, 90, 110] and 3D object detection [63, 64, 77, 98, 112, 112, 114].”
Campus3D: A Photogrammetry Point Cloud Benchmark for Hierarchical Understanding of Outdoor Scene (2020, methodology): “Forinstancesegmentation,weappliedcoverage(Cov)andweighted coverage (WCov) introduced into 3D tasks by Wang and Jia [2], with the definition as eq.”
Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion (2020, methodology): “(Wang et al. 2019b), Pham et al .”
An End-to-End Transformer Model for 3D Object Detection (2021, methodology): “We focus on methods that directly use 3D point clouds [40, 51, 75, 85].”
Review: deep learning on 3D point clouds (2020, background): “Some state-of-the-art works on instance segmentation on point cloud are [89, 90, 91, 92, 93] which are built on PointNet/PointNet++ feature learning backbone.”
Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds (2019, methodology): “ASIS [51], JSIS3D [34], MASC [30], 3D-BEVIS [8] and [28] use the same strategy to group point-level features for instance segmentation.”
OccuSeg: Occupancy-Aware 3D Instance Segmentation (2020, methodology): “ASIS [42] network(GPU): 650 mean shift(CPU): 53886 block merging(CPU): 2221 56757”