Multi-Modal Fusion Transformer for End-to-End Autonomous Driving

By Aditya Prakash, Kashyap Chitta, Andreas Geiger. Computer Vision and Pattern Recognition, 2021

750 citations recorded by the source corpus.

How should representations from complementary sensors be integrated for autonomous driving? Geometry-based sensor fusion has shown great promise for perception tasks such as object detection and motion forecasting. However, for the actual driving task, the global context of the 3D scene is key, e.g. a change in traffic light state can affect the behavior of a vehicle geometrically distant from that traffic light. Geometry alone may therefore be insufficient for effectively fusing representations in end-to-end driving models. In this work, we demonstrate that imitation learning policies based on existing sensor fusion methods under-perform in the presence of a high density of dynamic agents and complex scenarios, which require global contextual reasoning, such as handling traffic oncoming from multiple directions at uncontrolled intersections. Therefore, we propose TransFuser, a novel Multi-Modal Fusion Transformer, to integrate image and LiDAR representations using attention. We experimentally validate the efficacy of our approach in urban settings involving complex scenarios using the CARLA urban driving simulator. Our approach achieves state-of-the-art driving performance while reducing collisions by 76% compared to geometry-based fusion.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c0559fc7e7d6ea0f783ba791ddd5deaa74cf58a9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Planning-oriented Autonomous Driving (2022, background): “Pioneering works [14,16,21,22,78,95, 97, 106] verified this vanilla design in the closed-loop simulation [26].”
VAD: Vectorized Scene Representation for Efficient Autonomous Driving (2023, methodology): “3.4).”
6G for Vehicle-to-Everything (V2X) Communications: Enabling Technologies, Challenges, and Opportunities (2020, background): “As far as vision-based ML is concerned, multi-modal reasoning by fusing camera frames and LiDAR scans is investigated in [132] for better object detection to help autonomous driving.”
Parting with Misconceptions about Learning-based Vehicle Motion Planning (2023, methodology): “Ego-forecasting approaches include both end-to-end methods that utilize LiDAR scans [40, 41], RGB images [42, 43, 44, 45, 14, 46] or both [13, 47, 5], as well as modular methods involving lower-dimensional inputs like bird’s eye view (BEV) grids or state vectors [26, 48, 11, 49, 50].”
Image Dehazing Transformer with Transmission-Aware 3D Position Embedding (2022, background): “Based on its strong capability of modeling long-range dependencies by stacked self-attention and feed-forward, it has inspired the computer vision community to investigate how to apply Transformer in related tasks such as object detection [5], image segmentation [31], and autonomous driving [23].”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, methodology): “Attention has been explored in the context of driving for lane changing [65], object detection [31], [66], [67], [68], motion forecasting [31], [69], [70], [71], [72], [73], [74], [75], [76], [77], driver attention prediction [78], [79] and recently also for end-to-end driving [39], [41].”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, methodology): “It is less discussed due to its inferior performance [6], [141].”
Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer (2022, methodology): “Benchmark We evaluated our methods on three benchmarks: CARLA public leaderboard [44], the Town05 benchmark [8] and CARLA 42 Routes benchmark [17].”
LMDrive: Closed-Loop End-to-End Driving with Large Language Models (2023, methodology): “Compared with the previous CARLA benchmark, Town05 [33] and Longest6 [8] that navigate the agent with discrete driving commands or target waypoints, our benchmark only provides the AV with a navigation instruction and optional notice instructions in natural language.”
A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation (2024, methodology): “For instance, TransFuser was developed in [56] to combine feature maps using multiple transformers after each convolutional block.”
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers (2022, methodology): “In Table VII, we compare CMX with other models dedicated to RGB-LiDAR data fusion, including PMF [14] and Trans-Fuser [104].”
DriveGPT4: Interpretable End-to-End Autonomous Driving Via Large Language Model (2023, background): “To better integrate and optimize the entire system, some approaches pro-pose training the network in an end-to-end manner, eliminating the need for discontinuous intermediate steps [8], [9], [10].”
V2V4Real: A Real-World Large-Scale Dataset for Vehicle-to-Vehicle Cooperative Perception (2023, positive): “, first detect the object in 2D images, then use the obtained information to further process point clouds; more recent works [21,28] develop end-to-end fusion pipelines and leverage cross-attention [38] to perform feature alignment.”
VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning (2024, background): “Existing learning-based planning methods [23, 19, 21, 40, 16, 54] follow a deterministic paradigm to directly regress the action.”
A Survey of Visual Transformers (2021, methodology): “TransFuser [129], COTR [130], and mmFormer [134] deploy a hybrid model.”
Trajectory-guided Control Prediction for End-to-end Autonomous Driving: A Simple yet Strong Baseline (2022, methodology): “The output prediction of the model for endto-end autonomous driving generally falls into two forms: trajectory/waypoints [4, 11, 43, 15, 28, 10] and direct control actions [17, 37, 18, 40, 12, 55, 9].”
DriveLM: Driving with Graph Visual Question Answering (2023, neutral): “Directly translating images to motion as in [12, 54] is extremely challenging.”
nuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles (2021, methodology): “Multi-modal methods have also been explored in recent works [16, 20, 13].”