KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D

By Yiyi Liao, Jun Xie, Andreas Geiger. 2021

956 citations recorded by the source corpus.

For the last few decades, several major subfields of artificial intelligence including computer vision, graphics, and robotics have progressed largely independently from each other. Recently, however, the community has realized that progress towards robust intelligent systems such as self-driving cars requires a concerted effort across the different fields. This motivated us to develop KITTI-360, successor of the popular KITTI dataset. KITTI-360 is a suburban driving dataset which comprises richer input modalities, comprehensive semantic instance annotations and accurate localization to facilitate research at the intersection of vision, graphics and robotics. For efficient annotation, we created a tool to label 3D scenes with bounding primitives and developed a model that transfers this information into the 2D image domain, resulting in over 150k images and 1B 3D points with coherent semantic instance annotations across 2D and 3D. Moreover, we established benchmarks and baselines for several tasks relevant to mobile perception, encompassing problems from computer vision, graphics, and robotics on the same dataset, e.g., semantic scene understanding, novel view synthesis and semantic SLAM. KITTI-360 will enable progress at the intersection of these research areas and thus contribute towards solving one of today's grand challenges: the development of fully autonomous self-driving systems.

Subfields: Computer Vision and Pattern Recognition, Environmental Engineering, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Environmental Engineering

In Evaluation & Benchmarks

Citations in the corpus

Projected GANs Converge Faster (2021, methodology): “We compare on LSUN cat and horse [84], ADE indoor (a subset of ADE [91] proposed in [3]), the full Oxford ﬂowers dataset with 8k images [56], KITTI-ﬁsheye (a subset of KITTI-360 [46], consisting of ﬁsheye images), STL-10 [8], CUB200 [80], Stanford Dogs [36], and Stanford Cars [42].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, neutral): “According to the GLO authors, this method can be thought of as a Discriminator-less GAN.”
OpenScene: 3D Scene Understanding with Open Vocabularies (2022, background): “These works have yielded network architectures and training protocols that have significantly pushed the boundary of several 3D scene understanding benchmarks, including 3D object classification [56], 3D object detection and localization [4, 6, 15, 49], 3D semantic and instance segmentation [2, 3, 5”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, methodology): “Results on novel view color and semantic synthesis tasks on KITTI-360 [30] dataset.”
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers (2022, methodology): “3% on KITTI-360 (RGB-L) [25] datasets.”
UniSim: A Neural Closed-Loop Sensor Simulator (2023, neutral): “Similar to other NVS benchmarks [41], we subsample the sensor data by two, training on every other frame and testing on the remaining frames, dubbed “interpolation” test.”
DanceTrack: Multi-Object Tracking in Uniform Appearance and Diverse Motion (2021, background): “More recently, BDD100K [36], Waymo [28] and KITTI360 [18] are made available to the public, still focusing on autonomous driving scenarios but providing much larger scale data than KITTI.”
MonoScene: Monocular 3D Semantic Scene Completion (2021, neutral): “Cityscapes [17] Nuscenes [5] SemanticKITTI [3] KITTI-360 [44]”
Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving (2023, neutral): “Compared to conventional datasets such as SemanticKITTI [2] and KITTI-360 [24], our Occ3D is the first dataset to offer the surround-view images and high-resolution 3D voxel occupancy representation with the most diverse scenarios.”
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, positive): “For the KITTI-360 [7] dataset, we only use sequential images from a single camera as input, with a resolution of 1408 × 376 .”
Urban Radiance Fields (2021, background): “Neither system leverages lidar data, which is available in most outdoor mapping platforms [6, 7, 20, 22, 25, 32, 36, 61, 63] nor do they attempt to extract 3D surface reconstructions.”