SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences

By Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, Juergen Gall. IEEE International Conference on Computer Vision, 2019

2,459 citations recorded by the source corpus.

Semantic scene understanding is important for various applications. In particular, self-driving cars need a fine-grained understanding of the surfaces and objects in their vicinity. Light detection and ranging (LiDAR) provides precise geometric information about the environment and is thus a part of the sensor suites of almost all self-driving cars. Despite the relevance of semantic scene understanding for this application, there is a lack of a large dataset for this task which is based on an automotive LiDAR. In this paper, we introduce a large dataset to propel research on laser-based semantic segmentation. We annotated all sequences of the KITTI Vision Odometry Benchmark and provide dense point-wise annotations for the complete 360-degree field-of-view of the employed automotive LiDAR. We propose three benchmark tasks based on this dataset: (i) semantic segmentation of point clouds using a single scan, (ii) semantic segmentation using multiple past scans, and (iii) semantic scene completion, which requires to anticipate the semantic scene in the future. We provide baseline experiments and show that there is a need for more sophisticated models to efficiently tackle these tasks. Our dataset opens the door for the development of more advanced methods, but also provides plentiful data to investigate new research directions.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Environmental Engineering

View Paper: https://www.semanticscholar.org/paper/4511f4100decc138031f93212cfd921bf42f72e2

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Environmental Engineering

Citations in the corpus

Hydra: A Real-time Spatial Perception System for 3D Scene Graph Construction and Optimization (2022, neutral): “The literature has focused on both object-based maps [10, 14, 41, 42, 53, 56] and dense maps, including volumetric models [23, 38, 40], point clouds [6, 32, 61], and 3D meshes [48, 51].”
RELLIS-3D Dataset: Data, Benchmarks and Analysis (2020, methodology): “Although imbalanced class distributions exist in almost all current available urban datasets, the overall class distributions across distinct urban datasets are quite similar [23], [18].”
LiDARsim: Realistic LiDAR Simulation by Leveraging the Real World (2020, methodology): “Note that these annotations are prevalent in existing benchmarks such as KITTI or Nuscenes[13, 5].”
Kimera: an Open-Source Library for Real-Time Metric-Semantic Localization and Mapping (2019, background): “A sparser set of contributions address other sensing modalities, including monocular cameras ( e.g., CNN-SLAM [40], VSO [41], VITAMIN-E [42], XIVO [31]) and lidar ( e.g., SemanticKitti [43], SegMap [30]).”
Campus3D: A Photogrammetry Point Cloud Benchmark for Hierarchical Understanding of Outdoor Scene (2020, methodology): “Firstly, the datasets may face various limitations from their sources which are are either RGB-D images [1, 9, 10, 28] or light detection and ranging (LiDAR) based mobile laser scanning (MLS) [3, 25, 26, 31] and terrestrial laser scanning (TLS) [14].”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “It is a core computer vision task that could be beneficial to various real-world applications such as medical image analysis [209], [210], [211], [212] and autonomous driving [213], [214], [215].”
RPVNet: A Deep and Efficient Range-Point-Voxel Fusion Network for LiDAR Point Cloud Segmentation (2021, methodology): “In this experiment, we compare the results of our proposed method with existing state-of-the-art LiDAR segmentation methods on SemanticKITTI test set.”
SqueezeSegV3: Spatially-Adaptive Convolution for Efficient Point-Cloud Segmentation (2020, methodology): “Following [1], sequences-{0-7} and {9, 10} (19130 scans) are used for training, sequence-08 (4071 scans) is for validation, and sequences-{11-21} (20351 scans) are for test.”
Deep Learning for 3D Point Clouds: A Survey (2019, background): “the KITTI Vision Benchmark Suite [14], [15].”
RangeNet ++: Fast and Accurate LiDAR Semantic Segmentation (2019, methodology): “We train and evaluate our approach on a largescale dataset that provides dense point-wise annotations for the entire KITTI Odometry Benchmark [7], [1].”
VoxFormer: Sparse Voxel Transformer for Camera-Based 3D Semantic Scene Completion (2023, methodology): “We verify VoxFormer on SemanticKITTI [5], which provides dense semantic annotations for each LiDAR sweep from the KITTI Odometry Benchmark [71] composed of 22 outdoor driving scenarios.”
(AF)2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network (2021, methodology): “Figure 1: Comparison of our proposed method with Sal-saNext [9] and MinkNet42 [8] on SemanticKITTI benchmark [3].”
OpenGAN: Open-Set Recognition via Open Data Generation (2021, neutral): “Interestingly, classic semantic segmentation benchmarks evaluate these “other” pixels as a separate background class [16], but Cityscapes ignores them in its evaluation (as do many other contemporary datasets [8, 3, 42, 34]).”
Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving (2023, positive): “Compared to conventional datasets such as SemanticKITTI [2] and KITTI-360 [24], our Occ3D is the first dataset to offer the surround-view images and high-resolution 3D voxel occupancy representation with the most diverse scenarios.”
3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans (2020, background): “clouds [8, 22, 60, 97], meshes [30, 85, 88], surfels [101, 104], or volumetric models [30, 67, 70].”
Semantic Segmentation on Swiss3DCities: A Benchmark Study on Aerial Photogrammetric 3D Pointcloud Dataset (2020, methodology): “Most existing datasets [27], [3], [32] for outdoor 3D semantic segmentation are motivated by real-time autonomous driving applications, and are therefore acquired at low resolution by street-level Light Detection and Ranging (LiDAR) sensors; this yields incomplete point clouds (for example, areas fa”
Lidar for Autonomous Driving: The Principles, Challenges, and Trends for Automotive Lidar and Perception Systems (2020, background): “Due to a lack of massive annotated data sets, the performances of these two methods are not ready for deployment in real situations, although this situation has been changed by SemanticKITTI [54], [62], the latest and biggest pointwise annotated data set based on KITTI.”
Improving Multispectral Pedestrian Detection by Addressing Modality Imbalance Problems (2020, background): “Recent years have witnessed increasing researches towards object detection among vision community by taking the advantages of multi-modal inputs, such as RGB + thermal, RGB + depth, RGB + LiDAR and so on [19,17,15,1].”