Multi-level Fusion Based 3D Object Detection from Monocular Images

By Bin Xu, Zhenzhong Chen. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

330 citations recorded by the source corpus.

In this paper, we present an end-to-end multi-level fusion based framework for 3D object detection from a single monocular image. The whole network is composed of two parts: one for 2D region proposal generation and another for simultaneously predictions of objects' 2D locations, orientations, dimensions, and 3D locations. With the help of a stand-alone module to estimate the disparity and compute the 3D point cloud, we introduce the multi-level fusion scheme. First, we encode the disparity information with a front view feature representation and fuse it with the RGB image to enhance the input. Second, features extracted from the original input and the point cloud are combined to boost the object detection. For 3D localization, we introduce an extra stream to predict the location information from point cloud directly and add it to the aforementioned location prediction. The proposed algorithm can directly output both 2D and 3D object detection results in an end-to-end fashion with only a single RGB image as the input. The experimental results on the challenging KITTI benchmark demonstrate that our algorithm significantly outperforms monocular state-of-the-art methods.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/bf2793fc09176f8bf23b3a2b3c6b32185e8a8329

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Probabilistic and Geometric Depth: Detecting Objects in Perspective (2021, methodology): “Previous methods [10, 11] first use an extra cumbersome depth estimation model to complement 2D detectors on depth information.”
Accurate Monocular 3D Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving (2019, methodology): “Here five methods are chosen for comparisons: Mono3D [3], Deep3DBox [23] and Multi-Fusion [37], ROI-10D [20], MonoGRNet [30] and Pseudo-LiDAR [31].”
Pseudo-LiDAR++: Accurate Depth for 3D Object Detection in Autonomous Driving (2019, methodology): “…pipeline (Ren et al., 2015; He et al., 2017; Lin et al., 2017) — fell far behind in localizing objects in 3D (Li et al., 2019a; Xiang et al., 2015; 2017; Chabot et al., 2017; Mousavian et al., 2017; Chen et al., 2015; Xu & Chen, 2018; Chen et al., 2016; Pham & Jeon, 2017; Chen et al., 2018)3.”
Stereo R-CNN Based 3D Object Detection for Autonomous Driving (2019, methodology): “Instead of directly using the depth input [4, 27] which does not explicitly utilize the object property, we treat the object RoI as an entirety rather than independent pixels.”
GS3D: An Efficient 3D Object Detection Framework for Autonomous Driving (2019, methodology): “Among the previous works, [24, 19] use val1, and [2, 3] use val2, and [1, 27] use them both.”
SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation (2020, methodology): “A common approach is to apply an additional 3D network branch to regress orientation and translation of object instances, see [3, 23, 37, 19, 14, 25, 22, 31].”
nuScenes: A Multimodal Dataset for Autonomous Driving (2019, background): “However, 3D localization from images is challenging [13, 12, 57, 80, 69, 66, 73].”
BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022, methodology): “For the 3D object detection task, early methods are similar to 2D detection methods [1, 28, 48, 39, 52], which usually predict the 3D bounding boxes based on 2D bounding boxes.”
M3D-RPN: Monocular 3D Region Proposal Network for Object Detection (2019, methodology): “64) on the moderate and hard settings despite being trained as a shared multi-class model and compared to single model methods [7,8, 28,40].”
Cross-view Transformers for real-time Map-view Semantic Segmentation (2022, methodology): “Psuedo-lidar based approaches [24, 44, 45, 46] project to 3D points using a depth estimate and leverage 3D point based architectures (e.g. [18, 30, 43]) with 2D labels.”
Learning Depth-Guided Convolutions for Monocular 3D Object Detection (2019, methodology): “Previous monocular 3D detection methods [36, 26, 41, 1, 19, 17, 4, 54] usually make assumptions about the scene geometry and use this as a constraint to train the 2D-to-3D mapping.”
Multi-Task Multi-Sensor Fusion for 3D Object Detection (2019, background): “Over the past few years we have seen a plethora of methods that tackle the problem of 3D object detection from monocular images [2, 31], stereo cameras [4] or LiDAR point clouds [36, 34, 16].”
LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving (2019, background): “Some approaches [4, 3, 20, 25] have tried to tackle 3D detection without using range data from LiDAR.”
Pseudo-LiDAR From Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving (2018, methodology): “[5, 6, 22, 30] apply stereo-based depth estimation to obtain the true 3D coordinates of each pixel.”
Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, methodology): “Besides, we add results from VoxelNet [7] and MonoFusion [8] for comparison.”
RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving (2020, methodology): “In comparison, monocular camera devices are cheaper and convenient which makes it drawing an increasing attention in many application scenarios [7, 28, 42].”
Objects are Different: Flexible Monocular 3D Object Detection (2021, neutral): “Considering the difficulty in perceiving 3D environments from 2D images, most existing methods for monocular 3D object detection utilize extra information to simplify the task, which includes pretrained depth estimation modules [30, 45, 46, 47], annotated keypoints [2] and CAD models [32].”
MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships (2020, methodology): “An alternative family of methods is to predict a stand-alone depth or disparity information of the monocular image at the first stage [25, 26, 38, 41].”