Multimodal End-to-End Autonomous Driving

By Yi Xiao, Felipe Codevilla, Akhil Gurram, Onay Urfalioglu, Antonio M. López. IEEE transactions on intelligent transportation systems (Print), 2019

322 citations recorded by the source corpus.

A crucial component of an autonomous vehicle (AV) is the artificial intelligence (AI) is able to drive towards a desired destination. Today, there are different paradigms addressing the development of AI drivers. On the one hand, we find modular pipelines, which divide the driving task into sub-tasks such as perception and maneuver planning and control. On the other hand, we find end-to-end driving approaches that try to learn a direct mapping from input raw sensor data to vehicle control signals. The later are relatively less studied, but are gaining popularity since they are less demanding in terms of sensor data annotation. This paper focuses on end-to-end autonomous driving. So far, most proposals relying on this paradigm assume RGB images as input sensor data. However, AVs will not be equipped only with cameras, but also with active sensors providing accurate depth information (e.g., LiDARs). Accordingly, this paper analyses whether combining RGB and depth modalities, i.e. using RGBD data, produces better end-to-end AI drivers than relying on a single modality. We consider multimodality based on early, mid and late fusion schemes, both in multisensory and single-sensor (monocular depth estimation) settings. Using the CARLA simulator and conditional imitation learning (CIL), we show how, indeed, early fusion multimodality outperforms single-modality.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a30cce762c2ebf1cd13bd091c57d514db7094c47

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2021, background): “, depth sensors, etc) [81] to perceive his surroundings.”
A2D2: Audi Autonomous Driving Dataset (2020, methodology): “This allows A2D2 to be used for more ﬁelds of research in autonomous driving, e.g . end-to-end learning as in [16] and [17, 18] (synthetic data).”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, background): “Imitation Learning (IL).”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, methodology): “[62] employed inputs consisting of a monocular RGB image from a forward-facing camera and the vehicle speed.”
Multi-Modal Fusion Transformer for End-to-End Autonomous Driving (2021, background): “Multi-Modal Autonomous Driving: Recent multi-modal methods for end-to-end driving [58, 65, 51, 3] have shown that complementing RGB images with depth and semantics has the potential to improve driving performance.”
Domain-specific Genetic Algorithm for Multi-tenant DNNAccelerator Scheduling (2021, background): “First, end applications (such as AR/VR [55] or self-driving [16, 117]) and cloud services (such as search [36, 124] and translation [58, 62]) often rely on several DNN models internally.”
Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline (2022, background): “Diverse optimized approaches are presented based on them, such as multi-modal inputs [22, 53], multi-task learning [56, 37, 24, 27, 31, 26, 63], dataset aggregation [45] and knowledge distillation [61, 60].”
nuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles (2021, background): “[20] evaluates its planner using off-road time, off-lane time and number of crashes, while [13, 22] report the success rate of reaching a given destination within a fixed time window.”
DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language Model (2023, background): “End-to-end Autonomous Driving: End-to-end autonomous driving aims to directly predict the vehicle path and low-level control signals based on visual inputs [8], [9], [10], [29], [30].”
A Survey of End-to-End Driving: Architectures and Training Methods (2020, methodology): “1) Navigational commands: Navigating from point A to point B can be achieved by providing an additional navigational command assuming values like “go left”, “go right”, “go straight” and ”follow the road” [6], [11], [24], [56], [66], [84], [87].”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, unknown): “In [257], a simulation framework is proposed to apply importance-sampling strategies to accelerate the evaluation of rare-event probabilities.”