Accurate Monocular 3D Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving

By Xinzhu Ma, Zhihui Wang, Haojie Li, Wanli Ouyang, Pengbo Zhang. 2019

342 citations recorded by the source corpus.

In this paper, we propose a monocular 3D object detection framework in the domain of autonomous driving. Unlike previous image-based methods which focus on RGB feature extracted from 2D images, our method solves this problem in the reconstructed 3D space in order to exploit 3D contexts explicitly. To this end, we first leverage a stand-alone module to transform the input data from 2D image plane to 3D point clouds space for a better input representation, then we perform the 3D detection using PointNet backbone net to obtain objects’ 3D locations, dimensions and orientations. To enhance the discriminative capability of point clouds, we propose a multi-modal feature fusion module to embed the complementary RGB cue into the generated point clouds representation. We argue that it is more effective to infer the 3D bounding boxes from the generated 3D scene space (i.e., X,Y, Z space) compared to the image plane (i.e., R,G,B image plane). Evaluation on the challenging KITTI dataset shows that our approach boosts the performance of state-of-the-art monocular approach by a large margin.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

RTM3D: Real-time Monocular 3D Detection from Object Keypoints for Autonomous Driving (2020, methodology): “Real Time Stereo Depth Shape/CAD Segmentation Mono3D [7] 3DOP [8], stereoRCNN [22] MF3D [42], Peseudo-LiDAR [39], MonoPSR [17] AM3D[27] Mono3D++ [14],Deep-MENTA [6], 3DVP [40] Deep3DBox [28],GS3D [20],MonoGRNet [33], FQNet[25], M3D-RPN[4] Shift-RCNN [30] Ours(RTM3D) Image-only in Monocular 3D Object”
Delving into Localization Errors for Monocular 3D Object Detection (2021, methodology): “In contrast, the accuracy of 3D detection from only monocular images [35, 2, 10, 25, 24, 12] is obviously lower than that from LiDAR or stereo.”
SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, methodology): “Depth-based methods explicitly predict depth maps for image inputs to extract 3D geometric information of the scene before perception [54, 36, 44, 39, 28, 35, 30, 42, 21, 65].”
Geometry Uncertainty Projection Network for Monocular 3D Object Detection (2021, methodology): “Existing methods focus on deep representation learning [39] and geometry priors [29, 30, 46, 56].”
Cross-view Transformers for real-time Map-view Semantic Segmentation (2022, background): “Psuedo-lidar based approaches [24, 43, 44, 45] project to 3D points using a depth estimate and leverage 3D point based architectures (e.”
Invisible for both Camera and LiDAR: Security of Multi-Sensor Fusion based Perception in Autonomous Driving Under Physical-World Attacks (2021, background): “Camera images, on the other hand, cannot directly provide the depth information of an object [19], [25], which can be overcome by LiDARs.”
Categorical Depth Distribution Network for Monocular 3D Object Detection (2021, background): “To combat this effect, monocular object detection methods [13, 36, 37, 59] often learn depth explicitly, by training a monocular depth estimation network in a separate stage.”
EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection (2020, positive): “, the KITTI dataset [6] and SUN-RGBD dataset [33]. 2 Related Work 3D object detection based on camera images. Recent 3D object detection methods pay much attention to camera images, such as monocular [23,29,12,15,20] and stereo images [16,35]. Chen et al. [1] obtain 2D bounding boxes with a CNN-base”
SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation (2020, background): “Other approaches, like [21, 34, 9], generate hand-crafted features by transforming region of interest on images to other representations.”
MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships (2020, methodology): “An alternative family of methods is to predict a stand-alone depth or disparity information of the monocular image at the first stage [25, 26, 38, 41].”
Learning Depth-Guided Convolutions for Monocular 3D Object Detection (2019, methodology): “Recent methods towards the above goal can be generally categorized into two streams as image-based approaches [36, 26, 41, 19, 17, 4] and pseudo-LiDAR point-based approaches [48, 33, 50].”
Is Pseudo-Lidar needed for Monocular 3D Object detection? (2021, methodology): “As the RoI contains both objects as well as background pixels, the resulting features are ﬁltered via foreground masks computed based on the associated RoI depth map [42].”