FUTR3D: A Unified Sensor Fusion Framework for 3D Detection

By Xuanyao Chen, Tianyuan Zhang, Yue Wang, Yilun Wang, Hang Zhao. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2022

381 citations recorded by the source corpus.

Sensor fusion is an essential topic in many perception systems, such as autonomous driving and robotics. Existing multi-modal 3D detection models usually involve customized designs depending on the sensor combinations or setups. In this work, we propose the first unified end-to-end sensor fusion framework for 3D detection, named FUTR3D, which can be used in (almost) any sensor configuration. FUTR3D employs a query-based Modality-Agnostic Feature Sampler (MAFS), together with a transformer decoder with a set-to-set loss for 3D detection, thus avoiding using late fusion heuristics and post-processing tricks. We validate the effectiveness of our framework on various combinations of cameras, low-resolution LiDARs, high-resolution LiDARs, and Radars. On NuScenes dataset, FUTR3D achieves better performance over specifically designed methods across different sensor combinations. Moreover, FUTR3D achieves great flexibility with different sensor configurations and enables low-cost autonomous driving. For example, only using a 4-beam LiDAR with cameras, FUTR3D (58.0 mAP) surpasses state-of-the-art 3D detection model [41] (56.6 mAP) using a 32-beam LiDAR. Our code is available on the ${\text{project page}}$.

Subfields: Aerospace Engineering, Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/de4c059dfacfa993a3e688af140f854303882d7c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Occ3D: A Large-Scale 3D Occupancy Prediction Benchmark for Autonomous Driving (2023, neutral): “Several LiDAR-camera fusion methods are also proposed [35, 8, 28].”
DeepFusion: Lidar-Camera Deep Fusion for Multi-Modal 3D Object Detection (2022, background): “the features are combined after feature extraction [5,14,18].”
BEVFusion: A Simple and Robust LiDAR-Camera Fusion Framework (2022, background): “Despite a concurrent work [5] aims to fuse multi-modalities information in a single network, its design is limited to one speciﬁc detection head [53] while our framework can be generalized to arbitrary architectures.”
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, positive): “We compare our method with the latest SOTA approaches trained on the KITTI360, including NeRF-based PNF [6] and 3D Gaussian-based 3DGS [4].”
A Survey of Visual Transformers (2021, methodology): “D detection, FUTR3D [131] projects the object queries in the DETR-like decoder into a set of 3-”
BEVFusion: Multi-Task Multi-Sensor Fusion with Unified Bird's-Eye View Representation (2022, background): “Lately, FUTR3D [8] and TransFusion [1] deﬁne object queries in the 3D space and fuse image features onto these proposals.”
Unifying Voxel-based Representation with Transformer for 3D Object Detection (2022, background): “Compared with previous and even concurrent studies [8, 9], more key advances can be achieved with the proposed framework.”
VectorMapNet: End-to-end Vectorized HD Map Learning (2022, methodology): “Recently, several cross-view learning approaches (Philion & Fidler, 2020; Pan et al., 2020; Li et al., 2021; Zhou & Kr¨ahenb¨uhl, 2022; Wang et al., 2022; Chen et al., 2022) leverage the geometric structure of scenes to mitigate the mismatch between sensor inputs and BEV representations.”
3D Object Detection for Autonomous Driving: A Comprehensive Survey (2022, methodology): “Besides point cloud samplers, point cloud operators have also progressed rapidly, from the standard set abstraction [252, 339, 340, 342] to graph operators [256, 203] and Transformers [208].”
TransFuser: Imitation With Transformer-Based Sensor Fusion for Autonomous Driving (2022, neutral): “Most sensor fusion works consider perception tasks, e.g. object detection [14], [15], [16], [18], [19], [20], [21], [22], [23], [47], [48], [49], [50], [51], [52], [53], [54], [55], [56], [57], [58], [59], [60] and motion forecasting [24], [25], [26], [27], [28], [29], [30], [49], [61], [62].”