TransFusion: Robust LiDAR-Camera Fusion for 3D Object Detection with Transformers

By Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, Chiew-Lan Tai. Computer Vision and Pattern Recognition, 2022

996 citations recorded by the source corpus.

LiDAR and camera are two important sensors for 3D object detection in autonomous driving. Despite the increasing popularity of sensor fusion in this field, the robustness against inferior image conditions, e.g., bad illumination and sensor misalignment, is under-explored. Existing fusion methods are easily affected by such conditions, mainly due to a hard association of LiDAR points and image pixels, established by calibration matrices. We propose TransFusion, a robust solution to LiDAR-camera fusion with a soft-association mechanism to handle inferior image conditions. Specifically, our TransFusion consists of convolutional backbones and a detection head based on a transformer decoder. The first layer of the decoder predicts initial bounding boxes from a LiDAR point cloud using a sparse set of object queries, and its second decoder layer adaptively fuses the object queries with useful image features, leveraging both spatial and contextual relationships. The attention mechanism of the transformer enables our model to adaptively determine where and what information should be taken from the image, leading to a robust and effective fusion strategy. We additionally design an image-guided query initialization strategy to deal with objects that are difficult to detect in point clouds. TransFusion achieves state-of-the-art performance on large-scale datasets. We provide extensive experiments to demonstrate its robustness against degenerated image quality and calibration errors. We also extend the proposed method to the 3D tracking task and achieve the 1st place in the leader-board of nuScenes tracking, showing its effectiveness and generalization capability. [code release]

Subfields: Aerospace Engineering, Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/54fbb9300530d1deea596ad19807adedf1dc89dc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

1. Hongsheng Li
2. Song Han
3. Zhijian Liu
4. Shaoshuai Shi
5. Jiageng Mao
6. Heng Fan
7. Daniela Rus
8. Haotian Tang

In Artificial Intelligence

1. Song Han
2. Zhijian Liu
3. Hongsheng Li
4. Haotian Tang
5. Xiaogang Wang
6. Huizi Mao
7. Yue Liu
8. Xin Zuo

In Computer Vision and Pattern Recognition

1. Song Han
2. Hongsheng Li
3. Xiaogang Wang
4. Luc Van Gool
5. Yi Yang
6. Zhijian Liu
7. Dacheng Tao
8. Ming-Hsuan Yang

Citations in the corpus

• Multimodal Learning With Transformers: A Survey (2022, neutral): “DAR [215], [216], textual description & point cloud [31], acoustic & text [180], audio & visual observation for Audio-Visual Navigation [76], speech query & schema of SQL database [25],”
• A Survey of Visual Transformers (2021, neutral): “Camera-Based: MonoDTR [120], MonoDETR [121], DETR3D [122], TransFusion [123].”
• BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, methodology): “Following the same training settings [19, 56, 1], BEVFusion improves PointPillars and CenterPoint by 18.”
• BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, methodology): “We refer the readers to previous 3D detection papers [1, 67, 68] for more details.”
• A Multilevel Multimodal Fusion Transformer for Remote Sensing Semantic Segmentation (2024, background): “However, only a few works were devoted to the multimodal fusion tasks, emphasizing the fusion of multimodal information according to the characteristics of different modalities [14], [33], [34].”
• 3D Object Detection for Autonomous Driving: A Comprehensive Survey (2022, methodology): “59 - - - TransFusion [6] Fusion Intermediate 2022 - - - 68.”
• ICAFusion: Iterative Cross-Attention Guided Feature Fusion for Multispectral Object Detection (2023, background): “TransFusion [29] provides a robust solution for LiDAR-camera fusion with a soft-association mechanism to handle inferior image conditions.”
• TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, neutral): “Most sensor fusion works consider perception tasks, e.g. object detection [14], [15], [16], [18], [19], [20], [21], [22], [23], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60] and motion forecasting [24], [25], [26], [27], [28], [29], [30], [49], [61], [62].”