Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting

By Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes and 2 more. NeurIPS Datasets and Benchmarks, 2023

1,097 citations recorded by the source corpus.

We introduce Argoverse 2 (AV2) - a collection of three datasets for perception and forecasting research in the self-driving domain. The annotated Sensor Dataset contains 1,000 sequences of multimodal data, encompassing high-resolution imagery from seven ring cameras, and two stereo cameras in addition to lidar point clouds, and 6-DOF map-aligned pose. Sequences contain 3D cuboid annotations for 26 object categories, all of which are sufficiently-sampled to support training and evaluation of 3D perception models. The Lidar Dataset contains 20,000 sequences of unlabeled lidar point clouds and map-aligned pose. This dataset is the largest ever collection of lidar sensor data and supports self-supervised learning and the emerging task of point cloud forecasting. Finally, the Motion Forecasting Dataset contains 250,000 scenarios mined for interesting and challenging interactions between the autonomous vehicle and other actors in each local scene. Models are tasked with the prediction of future motion for"scored actors"in each scenario and are provided with track histories that capture object location, heading, velocity, and category. In all three datasets, each scenario contains its own HD Map with 3D lane and crosswalk geometry - sourced from data captured in six distinct cities. We believe these datasets will support new and existing machine learning research problems in ways that existing datasets do not. All datasets are released under the CC BY-NC-SA 4.0 license.

Subfields: Artificial Intelligence, Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/048056c0321876c0d582c3bf40b1883cda9260d5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “3 provides a comprehensive overview of the latest representative datasets for autonomous driving.”
Depth Anything 3: Recovering the Visual Space from Any Views (2025, positive): “We trained our metric depth model on 14 datasets, including Taskonomy [120], DIML (Outdoor) [15], DDAD [30], Argoverse [101], Lyft [], PandaSet [105], Waymo [84], ScanNet++ [117], ARK-itScenes [5], Map-free [3], DSEC [24], Driving Stereo [109] and Cityscapes [16] datasets.”
Query-Centric Trajectory Prediction (2023, methodology): “(i) While the flourishing forecasting models have achieved impressive performance on trajectory prediction benchmarks [7,13,49], today’s most advanced architectures specialized for this task [37, 38, 46, 56] fail to process the heterogeneous traffic scenes efficiently.”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, background): “The most popular datasets for this purpose include nuScenes [126], Argoverse [127], Waymo [128], and nuPlan [14].”
Motion Transformer with Global Intention Localization and Local Movement Refinement (2022, background): “The future behaviors of the agents highly depend on the agents’ interaction and road map.”
Metric3D v2: A Versatile Monocular Geometric Foundation Model for Zero-Shot Metric Depth and Surface Normal Estimation (2024, neutral): “12, including comic-style (Row 2) and CG(computer graphics)-generated objects (Row5)”
VectorMapNet: End-to-end Vectorized HD Map Learning (2022, methodology): “We further compare HDMapNet and VectorMapNet on Argoverse2 dataset, shown in Table 2.”
UniDepth: Universal Monocular Metric Depth Estimation (2024, methodology): “The training dataset utilized is the ensemble of Argoverse2 [43], Waymo [39], Driv-ingStereo [45], Cityscapes [6], BDD100K [49], Mapillary-PSD [1], A2D2 [18], ScanNet [7], and Taskonomy [51].”
3D Object Detection for Autonomous Driving: A Comprehensive Survey (2022, background): “Some datasets (Liao et al., 2021; Xiao et al., 2021; Geyer et al., 2020; Wilson et al., 2021; Caesar et al., 2020) can provide more fine-grained object classes, including animals, barriers, traffic cones, etc.”
Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Incorporating more datasets with action annotations [11, 112, 129] for collaborative training could be beneficial.”
Metric3D: Towards Zero-shot Metric 3D Prediction from A Single Image (2023, methodology): “Training Data DDAD [19] Outdoor LiDar ∼80K 36+ Lyft [24] Outdoor LiDar ∼50K 6+ Driving Stereo (DS) [62] Outdoor Stereo† ∼181K 1 DIML [9] Outdoor Stereo† ∼122K 10 Arogoverse2 [55] Outdoor LiDar ∼3515K 6+ Cityscapes [10] Outdoor Stereo† ∼170K 1 DSEC [16] Outdoor LiDar ∼26K 1 Mapillary PSD [34] Outdoor”
VoxelNeXt: Fully Sparse VoxelNet for 3D Object Detection and Tracking (2023, methodology): “VoxelNeXt presents promising results on large-scale datasets, including nuScenes [3], Waymo [45], and Argoverse2 [52].”
Why Braking? Scenario Extraction and Reasoning Utilizing LLM (2025, unknown): “To evaluate our proposed method, we annotate 700 data logs on the training set of the Argoverse 2 Sensor Dataset [6] with scenario labels.”