JSIS3D: Joint Semantic-Instance Segmentation of 3D Point Clouds With Multi-Task Pointwise Networks and Multi-Value Conditional Random Fields

By Quang-Hieu Pham, Dinh C. Nguyen, Binh-Son Hua, Gemma Roig, Sai-Kit Yeung. 2019

216 citations recorded by the source corpus.

Deep learning techniques have become the to-go models for most vision-related tasks on 2D images. However, their power has not been fully realised on several tasks in 3D space, e.g., 3D scene understanding. In this work, we jointly address the problems of semantic and instance segmentation of 3D point clouds. Specifically, we develop a multi-task pointwise network that simultaneously performs two tasks: predicting the semantic classes of 3D points and embedding the points into high-dimensional vectors so that points of the same object instance are represented by similar embeddings. We then propose a multi-value conditional random field model to incorporate the semantic and instance labels and formulate the problem of semantic and instance segmentation as jointly optimising labels in the field model. The proposed method is thoroughly evaluated and compared with existing methods on different indoor scene datasets including S3DIS and SceneNN. Experimental results showed the robustness of the proposed joint semantic-instance segmentation scheme over its single components. Our method also achieved state-of-the-art performance on semantic segmentation.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Environmental Engineering, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Environmental Engineering

In 3D Vision & Pose

Citations in the corpus

SoftGroup for 3D Instance Segmentation on Point Clouds (2022, methodology): “[29] present JSIS3D that incorporates the semantic and instance labels by a multi-value conditional random field model and jointly optimizes the labels to obtain object instances.”
JSNet: Joint Instance and Semantic Segmentation of 3D Point Clouds (2019, methodology): “In this task, we evaluate and compare our method with exist state-of-the-art methods including SGPN (Wang et al. 2018a), MT-PNet (Pham et al. 2019b ).”
Campus3D: A Photogrammetry Point Cloud Benchmark for Hierarchical Understanding of Outdoor Scene (2020, background): “ention-based models [41, 42]. Although most of these models are proposed for a single task, they can be involved in multitask learning. The examples are PointNet++ in ASIS [36] and PointNet in JSIS3D [22], which jointly learn instance embedding and semantic labeling in one structure. To jointly lear”
Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation (2020, unknown): “Mainstream methods [29, 36, 44, 41, 37, 24, 12, 50, 46, 38, 28, 30] of indoor point cloud segmentation learn the point features based on the raw point directly, which are often based on the pioneering work, i.e ., Point-Net, and promote the effectiveness of sampling, grouping and ordering to…”
Sparse Single Sweep LiDAR Point Cloud Segmentation via Learning Contextual Shape Priors from Scene Completion (2020, unknown): “(Pham et al. 2019) and Wei et al .”
Deep Learning for 3D Point Clouds: A Survey (2019, methodology): “Proposal-free methods [197], [198], [199], [200], [201], [202] do not have an object detection module.”
OccuSeg: Occupancy-Aware 3D Instance Segmentation (2020, methodology): “Based on the public datasets, our methods are quantitatively compared with a number of representative exist- ing methods, including SGPN [41], 3D-SIS [17], PanopticFusion [29], 3D-BoNet [46], MTML [21], ASIS [42] and JSIS3D [34].”
Review: deep learning on 3D point clouds (2020, background): “Some state-of-the-art works on instance segmentation on point cloud are [89, 90, 91, 92, 93] which are built on PointNet/PointNet++ feature learning backbone.”
An End-to-End Transformer Model for 3D Object Detection (2021, methodology): “We focus on methods that directly use 3D point clouds [40, 51, 75, 85].”
PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation (2020, methodology): “[33] develop a multi-task learning framework with a multi-value CRF model to jointly reason over both the semantic and instance labels.”
Self-Supervised Pretraining of 3D Features on any Point-Cloud (2021, methodology): “Scene-centric datasets can be broadly divided into indoor scens [5, 10, 18, 39, 58, 63, 75, 81, 84, 104], and outdoor (self-driving focussed) scenes [26, 71, 88].”
Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds (2019, methodology): “ASIS [51], JSIS3D [34], MASC [30], 3D-BEVIS [8] and [28] use the same strategy to group point-level features for instance segmentation.”