3D Bounding Box Estimation Using Deep Learning and Geometry

By Arsalan Mousavian, Dragomir Anguelov, John Flynn, Jana Kosecka. 2016

1,106 citations recorded by the source corpus.

We present a method for 3D object detection and pose estimation from a single image. In contrast to current techniques that only regress the 3D orientation of an object, our method first regresses relatively stable 3D object properties using a deep convolutional neural network and then combines these estimates with geometric constraints provided by a 2D object bounding box to produce a complete 3D bounding box. The first network output estimates the 3D object orientation using a novel hybrid discrete-continuous loss, which significantly outperforms the L2 loss. The second output regresses the 3D object dimensions, which have relatively little variance compared to alternatives and can often be predicted for many object types. These estimates, combined with the geometric constraints on translation imposed by the 2D bounding box, enable us to recover a stable and accurate 3D object pose. We evaluate our method on the challenging KITTI object detection benchmark [2] both on the official metric of 3D orientation estimation and also on the accuracy of the obtained 3D bounding boxes. Although conceptually simple, our method outperforms more complex and computationally expensive approaches that leverage semantic segmentation, instance level segmentation and flat ground priors [4] and sub-category detection [23][24]. Our discrete-continuous loss also produces state of the art results for 3D viewpoint estimation on the Pascal 3D+ dataset[26].

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Objects as Points (2019, methodology): “For 3D bounding box estimation, we regress to the object absolute depth, 3D bounding box dimensions, and object orientation [38].”
Towards Safe Autonomous Driving: Capture Uncertainty in the Deep Neural Network For Lidar 3D Vehicle Detection (2018, background): “Deep learning has been introduced to object detection in the autonomous driving settings that use cameras [1]–[3], Lidar [4]–[10], or both [11]–[17], and has set the benchmark on many popular datasets (e.”
nuScenes: A Multimodal Dataset for Autonomous Driving (2019, background): “However, 3D localization from images is challenging [13, 12, 57, 80, 69, 66, 73].”
FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection (2021, methodology): “To mention only a few, 3DOP [4] and MLFusion [35] use a depth estimation network, while Deep3DBox [21] uses a 2D object detector.”
3D-Aware Scene Manipulation via Inverse Graphics (2018, neutral): “We also compare with a 3D bounding box estimation method [Mousavian et al., 2017], which ﬁrst infers the object’s 2D bounding box and pose from input and then searches for its 3D bounding box.”
From Points to Parts: 3D Object Detection From Point Cloud With Part-Aware and Part-Aggregation Network (2019, background): “[34], [35] leveraged the geometry constraints between 3D and 2D bounding box to recover the 3D object pose.”
MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships (2020, methodology): “Therefore, there is a growing trend on performing monocular 3D object detection in research community in recent years [3, 5, 26, 27, 31, 36].”
PETRv2: A Unified Framework for 3D Perception from Multi-Camera Images (2022, neutral): “Previous works [2,6,13–15, 29, 35 , 37, 38 ] perform 3D object object”
Joint Monocular 3D Vehicle Detection and Tracking (2018, methodology): “Besides, our approach integrates with 2D detection and has the potential to jointly training, while [27] crops the input image with pre-computed boxes.”
YOLO3D: End-to-end real-time 3D Oriented Object Bounding Box Detection from LiDAR Point Cloud (2018, background): “ducing a discrete square grid. The Second one is the camera-only paradigm; which works by adding prior knowledge about the objects’ sizes, and trying to predict 3D bounding box using monocolor camera.[9] and [10] can produce highly accurate 3D bounding boxes using only camera images.[11] uses stereo”
Stereo R-CNN Based 3D Object Detection for Autonomous Driving (2019, methodology): “On the other hand, monocular camera provides alternative low-cost solutions[3, 21, 27] for 3D object detection.”
BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers (2022, methodology): “For the 3D object detection task, early methods are similar to 2D detection methods [1, 28, 48, 39, 52], which usually predict the 3D bounding boxes based on 2D bounding boxes.”
LaserNet: An Efficient Probabilistic 3D Object Detector for Autonomous Driving (2019, background): “Some approaches [4, 3, 20, 25] have tried to tackle 3D detection without using range data from LiDAR.”
DeepIM: Deep Iterative Matching for 6D Pose Estimation (2018, background): “As shown in Mousavian et al. (2017), the choice of reference frame is important to achieve good pose estimation results.”
Few-Shot Object Detection and Viewpoint Estimation for Objects in the Wild (2020, methodology): “Deep-learning methods for viewpoint estimation follow roughly three different paths: direct estimation of Euler angles [55, 50, 33, 24, 62, 63], template-based matching [20, 32, 51], and keypoint detection relying on 3D bounding box corners [39, 52, 13, 34, 36] or semantic keypoints [35, 65].”
What's Behind the Couch? Directed Ray Distance Functions (DRDF) for 3D Scene Reconstruction (2021, neutral): “For instance, in colorization [27, 70], where one is uncertain of the precise hue, the expected value averages the options, leading to brown re-sults; similar effects happen in rotation [7, 35, 43] and 3D estimation [31, 34, 65].”
Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation (2019, background): “On the other hand, category-level 3D object detection methods [43, 36, 9, 34, 49, 12] can estimate object class labels and 3D bounding boxes without requiring exact CAD models.”
DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion (2019, background): “[21] predicts 3D object parameters and recovers poses by single-view geometry constraints.”