nuScenes: A Multimodal Dataset for Autonomous Driving
By Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yuxin Pan, Giancarlo Baldan, Oscar Beijbom. 2019
8,536 citations recorded by the source corpus.
Robust detection and tracking of objects is crucial for the deployment of autonomous vehicle technology. Image based benchmark datasets have driven development in computer vision tasks such as object detection, tracking and segmentation of agents in the environment. Most autonomous vehicles, however, carry a combination of cameras and range sensors such as lidar and radar. As machine learning based methods for detection and tracking become more prevalent, there is a need to train and evaluate such methods on datasets containing range sensor data along with images. In this work we present nuTonomy scenes (nuScenes), the first dataset to carry the full autonomous vehicle sensor suite: 6 cameras, 5 radars and 1 lidar, all with full 360 degree field of view. nuScenes comprises 1000 scenes, each 20s long and fully annotated with 3D bounding boxes for 23 classes and 8 attributes. It has 7x as many annotations and 100x as many images as the pioneering KITTI dataset. We define novel 3D detection and tracking metrics. We also provide careful dataset analysis as well as baselines for lidar and image based detection and tracking. Data, development kit and more information are available online.
Subfields: Automotive Engineering, Computer Vision and Pattern Recognition, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Automotive Engineering
- Andreas Geiger
- Kashyap Chitta
- Pravendra Singh
- Pranav Singh Chib
- Kris M. Kitani
- Raquel Urtasun
- Hongyang Li
- Hongsheng Li
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Trevor Darrell
- Jitendra Malik
- Alan L. Yuille
- Andrew Zisserman
In Evaluation & Benchmarks
- Vladlen Koltun
- Jitendra Malik
- Luc Van Gool
- Trevor Darrell
- Chunhua Shen
- Sanja Fidler
- Kristen Grauman
- Vittorio Ferrari
Citations in the corpus
- SimpleClick: Interactive Image Segmentation with Simple Vision Transformers (2022, background): “Interactive image segmentation approaches have been widely applied to annotate large-scale image datasets, which drive the success of deep models in various applications, including video understanding [5,48], self-driving [7], and medical imaging [31, 40].”
- Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, background): “This has been recently achieved for autonomous driving [17,18,19,20,21,22,23,24] and scene understanding [25,26].”
- WoodScape: A Multi-Task, Multi-Camera Fisheye Dataset for Autonomous Driving (2019, background): “Task/Info Quantity KITTI [17] Cityscapes [10] Mapillary [39] nuScenes [6] ApolloScape [24] BDD100k [59] WoodScape Ours”
- Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming (2019, background): “…al., 2019, Bernuth et al., 2019, Hospach et al., 2016, Bernuth et al., 2018] or collecting real [Wen et al., 2015, Yu et al., 2018, Che et al., 2019, Caesar et al., 2019] and artificial [Gaidon et al., 2016, Ros et al., 2016, Richter et al., 2017, Johnson-Roberson et al., 2017] datasets with…”
- A Survey of Autonomous Driving: Common Practices and Emerging Technologies (2019, methodology): “Cityscapes [270] X X Vehicle Berkley DeepDrive [271] X X Vehicle X X Mapillary [272] X X Vehicle X X Oxford RobotCar [49] X X Vehicle X X X KITTI [174] X X X X Vehicle H3D [273] X X X Vehicle ApolloScape [274] X X X X Vehicle nuScenes [175] X X X X Vehicle X X Udacity [275] X X X X Vehicle DDD17 [85”
- Accelerating Reinforcement Learning with Learned Skill Priors (2020, background): “On the other hand, there is an abundance of collected agent experience available in domains like autonomous driving [3], indoor navigation [24], or robotic manipulation [4, 2].”
- A Study of Face Obfuscation in ImageNet (2021, background): “Frome et al. (2009) and Uittenbogaard et al. (2019) use blurring and inpainting to obfuscate faces and license plates in Google Street View. nuScenes (Caesar et al., 2020) is an autonomous driving dataset where faces and license plates are detected and then blurred.”
- PolarNet: An Improved Grid Representation for Online LiDAR Point Clouds Semantic Segmentation (2020, background): “While several large-scale LiDAR point clouds datasets are publicly available [9, 29, 42, 3], it is until recent that the semantic segmentation labels, provided by [1, 10], are able to match their scales.”
- PandaSet: Advanced Sensor Suite Dataset for Autonomous Driving (2021, background): “The nuScenes [1], Argoverse [17], Lyft L5 [18], Waymo Open [19], and A ∗ 3D datasets [5]”
- Depth Pro: Sharp Monocular Metric Depth in Less Than a Second (2024, neutral): “We report results on Booster (Ramirez et al., 2024), Middlebury (Scharstein et al., 2014), Sun-RGBD (Song et al., 2015), ETH3D (Sch¨ops et al., 2017), nuScenes (Caesar et al., 2020), and Sintel (Butler et al., 2012).”
- Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting (2023, methodology): “New datasets have also paved the way for new algorithms, with nuScenes [4], Lyft L5 [19], and the Waymo Open Motion Dataset [11] all releasing lane graphs after they proved to be essential in Argoverse 1 [6].”
- IoU Loss for 2D/3D Object Detection (2019, neutral): “This kind of representation is widely used in most of the popular 3D object detection benchmarks, such as KITTI [3] andnuScenes [23].”
- Large Scale Interactive Motion Forecasting for Autonomous Driving : The Waymo Open Motion Dataset (2021, methodology): “† nuScenes [4] provides annotations for 23 objects types (stationary vehicles are removed), but only vehicle is predicted.”
- ST3D: Self-training for Unsupervised Domain Adaptation on 3D Object Detection (2021, methodology): “We conduct experiments on four widely used autonomous driving datasets: KITTI [13], Waymo [38], nuSenses [4], and Lyft [20].”
- DETRs with Hybrid Matching (2022, neutral): “nuScenes [3].”
- ARKitScenes: A Diverse Real-World Dataset For 3D Indoor Scene Understanding Using Mobile RGB-D Data (2021, background): “For outdoor scene understanding, several large scale datasets with a variety of scenes in real scenarios were released that have powered deep learning algorithms [19, 13, 20, 21].”
- Opening up Open World Tracking (2021, background): “Existing MOT datasets focus on closed-set tracking on video [18, 24, 83, 90] or LiDAR streams [14, 75].”
- NuScenes-QA: A Multi-modal Visual Question Answering Benchmark for Autonomous Driving Scenario (2023, methodology): “The proposed NuScenes-QA is built upon nuScenes [5], which is a popular 3D perception dataset for autonomous driving.”