TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving

By Kashyap Chitta, Aditya Prakash, Bernhard Jaeger, Zehao Yu, Katrin Renz, Andreas Geiger. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022

755 citations recorded by the source corpus.

How should we integrate representations from complementary sensors for autonomous driving? Geometry-based fusion has shown promise for perception (e.g., object detection, motion forecasting). However, in the context of end-to-end driving, we find that imitation learning based on existing sensor fusion methods underperforms in complex driving scenarios with a high density of dynamic agents. Therefore, we propose TransFuser, a mechanism to integrate image and LiDAR representations using self-attention. Our approach uses transformer modules at multiple resolutions to fuse perspective view and bird's eye view feature maps. We experimentally validate its efficacy on a challenging new benchmark with long routes and dense traffic, as well as the official leaderboard of the CARLA urban driving simulator. At the time of submission, TransFuser outperforms all prior work on the CARLA leaderboard in terms of driving score by a large margin. Compared to geometry-based fusion, TransFuser reduces the average collisions per kilometer by 48%.

Subfields: Automotive Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1a8ac422f4e594155af1721837009264ce14fe32

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Automotive Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Vista: A Generalizable Driving World Model with High Fidelity and Versatile Controllability (2024, neutral): “Moreover, most of them are restricted to a specific control modality [54, 61, 90, 125], which hinders the accommodation to prevailing planning algorithms [12, 14, 21, 56, 58, 64] and extension to more applications like decision-making [127] or user interaction [74].”
Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer (2022, methodology): “Transfuser [8, 46] is an imitation learning method where the agent uses transformer to fuse information from the front camera image and LiDAR information.”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, methodology): “Restrictions apply. resources [22], [28], the ultimate necessity of explicit representations such as maps is uncertain.”
Parting with Misconceptions about Learning-based Vehicle Motion Planning (2023, methodology): “As a result, most current work on learned planning only compares to other learned methods, ignoring rule-based baselines [3, 5, 18].”
Planning-oriented Autonomous Driving (2022, background): “This is a popular practice in many domains, including general vision [79,92,108], autonomous driving 2 [15, 60, 101, 105], such as Transfuser [20], BEV-2 NMP [101] ✓ ✓ ✓ NEAT [19] ✓ ✓ BEVerse [105] ✓ ✓ ✓ (c.1) [14,16,78,97] ✓ (c.2) PnPNet † [57] ✓ ✓ ✓ ViP3D † [30] ✓ ✓ ✓ P3 [82] ✓ ✓ MP3 [11] ✓ ✓ ✓…”
DriveLM: Driving with Graph Visual Question Answering (2023, background): “Furthermore, unlike today’s AD models, humans do not navigate based on geometrically precise bird’s-eye view (BEV) representations [13, 26, 39].”
DiffusionDrive: Truncated Diffusion Model for End-to-End Autonomous Driving (2024, positive): “We begin from the representative deterministic end-to-end planner Transfuser [8] and turn it into a generative model Transfuser DP by simply replacing the regression MLP layers with the conditional diffusion model UNet following vanilla diffusion policy [7].”
LMDrive: Closed-Loop End-to-End Driving with Large Language Models (2023, methodology): “Compared with the previous CARLA benchmark, Town05 [33] and Longest6 [8] that navigate the agent with discrete driving commands or target waypoints, our benchmark only provides the AV with a navigation instruction and optional notice instructions in natural language.”
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking (2024, positive): “For instance, an extensive body of end-to-end approaches focuses on closed-loop simulators, utilizing single-frame cameras, LiDAR point clouds, or a combination of both for expert imitation [7, 11, 36, 8, 52, 43, 44, 12, 24, 57, 22].”
Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, methodology): “However, recent methodologies have improved interpretability in End-to-End models by generating auxiliary outputs [7, 13], attention maps [14, 15, 16, 9, 17, 18], and interpretable maps [18, 19, 8, 20, 12, 21].”