SEGCloud: Semantic Segmentation of 3D Point Clouds

By Lyne P. Tchapmi, Christopher Bongsoo Choy, Iro Armeni, JunYoung Gwak, Silvio Savarese. International Conference on 3D Vision, 2017

833 citations recorded by the source corpus.

3D semantic scene labeling is fundamental to agents operating in the real world. In particular, labeling raw 3D point sets from sensors provides fine-grained semantics. Recent works leverage the capabilities of Neural Networks (NNs), but are limited to coarse voxel predictions and do not explicitly enforce global consistency. We present SEGCloud, an end-to-end framework to obtain 3D point-level segmentation that combines the advantages of NNs, trilinear interpolation(TI) and fully connected Conditional Random Fields (FC-CRF). Coarse voxel predictions from a 3D Fully Convolutional NN are transferred back to the raw 3D points via trilinear interpolation. Then the FC-CRF enforces global consistency and provides fine-grained semantics on the points. We implement the latter as a differentiable Recurrent NN to allow joint optimization. We evaluate the framework on two indoor and two outdoor 3D datasets (NYU V2, S3DIS, KITTI, this http URL), and show performance comparable or superior to the state-of-the-art on all datasets.

Subfields: Aerospace Engineering, Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/131660f9f10a831968f2fafa0985f1b5eeb0b337

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Point-Voxel CNN for Efficient 3D Deep Learning (2019, methodology): “For example, SegCloud [40], SGPN [43], SPGraph [20], ParamConv [42], SSCN [7] and RSNet [14] are specialized in 3D semantic/instance segmentation.”
Point Transformer V3: Simpler, Faster, Stronger (2023, methodology): “Following a common practice [63, 75, 106], we withhold area 5 for testing and perform a 6-fold cross-validation.”
Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review (2020, methodology): “The CRFs have the advantage in combining low-level information such as the interactions between points to output multi-class inference for multi-class per-point labeling tasks, which compensates the fine local details that CNNs fail to capture.”
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, neutral): “Given a 3D scene as input, 3D semantic segmentation task aims to assign a semantic category to each point in the scene [2–4, 9, 14, 15, 22, 28, 29, 31, 38, 42, 44, 46, 47, 53, 54, 63, 64, 66, 68, 70, 73].”
SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation (2020, methodology): “Since a typical outdoor LiDAR (such as Velodyne HDL-64E) can collect about 200K points per scan, it is difficult for previous methods [22,38,47,26,37,31,29] to satisfy a real-time latency constraint.”
RandLA-Net: Efficient Semantic Segmentation of Large-Scale Point Clouds (2019, background): “For fair comparison, we only include the results of recently published strong baselines [4, 52, 53, 46, 69, 56, 26] and the current state-of-the-art approach KPConv [54].”
SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences (2019, methodology): “To overcome the limitations of the voxel-based representation, such as the exploding memory consumption when the resolution of the voxel grid increases, more recent approaches either upsample voxel-predictions [53] using a CRF or use different representations, like more efficient spatial subdivision”
PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation (2020, methodology): “Most current point cloud applications focus on general point clouds in which points are densely distributed on object surfaces, such as single 3D object shape recognition [34], indoor point cloud segmentation [31, 27], and reconstruction of outdoor scenes from point clouds [30].”
Spherical CNNs on Unstructured Grids (2019, background): “In the 3D learning literature, researchers have looked at 3D semantic segmentation on point clouds or voxels (Dai et al., 2017a; Qi et al., 2017a; Wang et al., 2018b; Tchapmi et al., 2017; Dai et al., 2017b).”
Learning Object Bounding Boxes for 3D Instance Segmentation on Point Clouds (2019, neutral): “Recent learning based approaches mainly include voxel-based [42; 46; 41; 23; 40; 11; 4] and point-based schemes [37; 19; 14; 16; 45].”
Point Transformer V2: Grouped Vector Attention and Partition-based Pooling (2022, methodology): “Following a common protocol [36, 4, 1], area 5 is withheld during training and used for testing.”
PointGroup: Dual-Set Point Grouping for 3D Instance Segmentation (2020, background): “Some other approaches [30, 43, 47, 39, 13, 7] align and voxelize point cloud to produce regular 3D ordered tensors for 3D convolution.”
PCT: Point cloud transformer (2020, background): “Inspired by strong progress made by convolutional neural networks (CNNs) in the field of image processing, many recent works [24, 17, 1, 31] have considered to define convolution operators that can aggregate local features for point clouds.”
DeepGCNs: Can GCNs Go As Deep As CNNs? (2019, background): “Several attempts in creating structure from 3D data exist by either representing it with multiple 2D views [35, 9, 3, 22], or by voxelization [5, 28, 32, 37].”
Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation (2020, methodology): “OccuSeg [13], SSCN [12] and SEGCloud [30] follow this line to utilize the voxel partition and apply regular 3D convolutions for LiDAR segmentation.”
Tangent Convolutions for Dense Prediction in 3D (2018, methodology): “An alternative way of increasing the accuracy of voxel-based techniques is to add differentiable postprocessing, modeled upon the dense CRF [26, 52].”