Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art

By Joel Janai, Fatma Güney, Aseem Behl, Andreas Geiger. 2017

934 citations recorded by the source corpus.

Recent years have witnessed enormous progress in AI-related fields such as computer vision, machine learning, and autonomous vehicles. As with any rapidly growing field, it becomes increasingly difficult to stay up-to-date or enter the field as a beginner. While several survey papers on particular sub-problems have appeared, no comprehensive survey on problems, datasets, and methods in computer vision for autonomous vehicles has been published. This book attempts to narrow this gap by providing a survey on the state-of-the-art datasets and techniques. Our survey includes both the historically most relevant literature as well as the current state of the art on several specific topics, including recognition, reconstruction, motion estimation, tracking, scene understanding, and end-to-end learning for autonomous driving. Towards this goal, we analyze the performance of the state of the art on several challenging benchmarking datasets, including KITTI, MOT, and Cityscapes. Besides, we discuss open problems and current research challenges. To ease accessibility and accommodate missing references, we also provide a website that allows navigating topics as well as methods and provides additional information.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Autonomous Driving: Common Practices and Emerging Technologies (2019, background): “Panoramic view is particularly desirable for applications such as navigation, localization and mapping [84].”
Semantic Foggy Scene Understanding with Synthetic Data (2017, neutral): “Cameras and the accompanying vision algorithms are widely used for applications such as surveillance [9], remote sensing [17], and automated cars [34], and their deployment keeps expanding.”
Solving Current Limitations of Deep Learning Based Approaches for Plant Disease Detection (2019, background): “Ever since its great breakthrough, DL [27] achieved has state-of-the-art performance, especially in the following domains: automatic game playing [28], natural language processing [29], autonomous vehicles [30], and much more.”
A Comparative Analysis of Object Detection Metrics with a Companion Open-Source Toolkit (2021, background): “Later on, advancements in object detection techniques led to their increasing adoption in areas that included the manufacturing industry with optical inspections [13], video surveillance [14], forensics [15,16], medical image analysis [17–19], autonomous vehicles [20], and trafﬁc monitoring [21].”
Autonomous Cars: Research Results, Issues, and Future Challenges (2019, background): “A more comprehensive study of computer vision in autonomous cars is presented in [47].”
Multimodal End-to-End Autonomous Driving (2019, background): “defined modules with particular responsibilities; for instance, to provide environmental perception [1], [2], as well as route planning and maneuver control [3], [4].”
Addressing Failure Prediction by Learning Model Confidence (2019, background): “Despite their growing success, safety remains a great concern when it comes to implement these models in real-world conditions [1, 19].”
Milestones in Autonomous Driving and Intelligent Vehicles: Survey of Surveys (2023, neutral): “[45], [46], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60], [61], [62], [63], [64], [65], [66], [67], scene understanding [68], [69], [70], tracking [71], [72],”
Depth-Attentional Features for Single-Image Rain Removal (2019, background): “, object detection [17] and tracking [40] for surveillance [2], autonomous driving [20], and driver assistance [33].”
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking (2024, neutral): “Autonomous vehicles (AVs) have gained immense research interest due to their potential to change transportation and improve traffic safety [23, 9].”
KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D (2021, background): “Only recently, the communities have realized that it is impossible to solve one without the other, e.g., in the context of self-driving [47].”
Planning and Decision-Making for Autonomous Vehicles (2018, background): “A recent survey (41) includes both historical and the current state-of-the-art literature on several specific topics, including recognition, reconstruction, motion estimation, tracking, scene understanding, and end-to-end learning on several benchmarking data sets, including the KITTI (42), ISPRS (I”
Deep Learning for LiDAR Point Clouds in Autonomous Driving: A Review (2020, methodology): “In [1], challenges, datasets, and methods in computer vision for AVs are reviewed.”
Legged Locomotion in Challenging Terrains using Egocentric Vision (2022, methodology): “This 264 technique has been fruitfully applied to self-driving [80, 81], autonomous drone flight [82, 83], 265 among others.”
Unsupervised Real-to-Virtual Domain Unification for End-to-End Highway Driving (2018, background): “Since the first successful demonstration in the 1980s [27, 10, 9], various methods have been investigated in the spectrum of vision-based driving models, from end-to-end methods to full pipeline methods [16].”
Class Similarity Weighted Knowledge Distillation for Continual Semantic Segmentation (2022, background): “Segmentation models are critical to many real-world applications, such as self-driving cars [1,17] and medical image diagnostics [14,40].”
A Survey on Multimodal Large Language Models for Autonomous Driving (2023, background): “Overall, it demonstrates the evolution and increasing sophistication of AD systems over several decades. offering state-of-the-art solutions for problems such as object detection, scene understanding, and vehicle localization [65,90,136].”
A Review and Comparative Study on Probabilistic Object Detection in Autonomous Driving (2020, background): “(especially deep learning) have been widely applied to safety-critical computer vision problems [2], eﬀorts to improve a network’s self-assessment ability, reliability and interpretability with uncertainty estimation are steadily increasing [3], [4].”