Safety-Enhanced Autonomous Driving Using Interpretable Sensor Fusion Transformer

By Hao Shao, Letian Wang, Ruobing Chen, Hongsheng Li, Yu Liu. Conference on Robot Learning, 2022

366 citations recorded by the source corpus.

Large-scale deployment of autonomous vehicles has been continually delayed due to safety concerns. On the one hand, comprehensive scene understanding is indispensable, a lack of which would result in vulnerability to rare but complex traffic situations, such as the sudden emergence of unknown objects. However, reasoning from a global context requires access to sensors of multiple types and adequate fusion of multi-modal sensor signals, which is difficult to achieve. On the other hand, the lack of interpretability in learning models also hampers the safety with unverifiable failure causes. In this paper, we propose a safety-enhanced autonomous driving framework, named Interpretable Sensor Fusion Transformer(InterFuser), to fully process and fuse information from multi-modal multi-view sensors for achieving comprehensive scene understanding and adversarial event detection. Besides, intermediate interpretable features are generated from our framework, which provide more semantics and are exploited to better constrain actions to be within the safe sets. We conducted extensive experiments on CARLA benchmarks, where our model outperforms prior methods, ranking the first on the public CARLA Leaderboard. Our code will be made available at https://github.com/opendilab/InterFuser

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a3335ad0d5e4061edda2b66567e517022642ea96

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hongsheng Li
2. Peng Gao
3. Renrui Zhang
4. Jiaming Han
5. Ziyu Guo
6. Xinggang Wang
7. Wenyu Liu
8. Qian Zhang

In Computer Vision and Pattern Recognition

1. Hongsheng Li
2. Xinggang Wang
3. Xiaogang Wang
4. Peng Gao
5. Wenyu Liu
6. Yu Liu
7. Qian Zhang
8. Luc Van Gool

Citations in the corpus

• Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, background): “In these years, 3D vision has gained significant attention and development, driven by the rising popularity of autonomous driving [11, 69, 71], navigation [72, 76, 98], 3D scene understanding [2, 43, 46, 74], and robotics [31, 67].”
• End-to-End Autonomous Driving: Challenges and Frontiers (2023, positive): “, Transformers [27]) to capture global context and representative features, as in TransFuser [6, 28] and many variants [29, 30, 31].”
• NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking (2024, neutral): “For instance, an extensive body of end-to-end approaches focuses on closed-loop simulators, utilizing single-frame cameras, LiDAR point clouds, or a combination of both for expert imitation [7, 11, 36, 8, 52, 43, 44, 12, 24, 57, 22].”
• LMDrive: Closed-Loop End-to-End Driving with Large Language Models (2023, methodology): “InterFuser [38] proposed a transformer-based framework to fully fuse and process information from multi-modal multi-view sensors for comprehensive scene understanding.”
• Recent Advancements in End-to-End Autonomous Driving Using Deep Learning: A Survey (2023, methodology): “However, recent methodologies have improved interpretability in End-to-End models by generating auxiliary outputs [7, 13], attention maps [14, 15, 16, 9, 17, 18], and interpretable maps [18, 19, 8, 20, 12, 21].”
• VADv2: End-to-End Vectorized Autonomous Driving via Probabilistic Planning (2024, positive): “Furthermore, compared to the previous best method [45] which only relies on cameras, VADv2 demonstrates even greater advantages, with a remarkable increase in Drive Score of up to 16.8.”