ROI-10D: Monocular Lifting of 2D Detection to 6D Pose and Metric Shape

By Fabian Manhardt, Wadim Kehl, Adrien Gaidon. Computer Vision and Pattern Recognition, 2018

302 citations recorded by the source corpus.

We present a deep learning method for end-to-end monocular 3D object detection and metric shape retrieval. We propose a novel loss formulation by lifting 2D detection, orientation, and scale estimation into 3D space. Instead of optimizing these quantities separately, the 3D instantiation allows to properly measure the metric misalignment of boxes. We experimentally show that our 10D lifting of sparse 2D Regions of Interests (RoIs) achieves great results both for 6D pose and recovery of the textured metric geometry of instances. This further enables 3D synthetic data augmentation via inpainting recovered meshes directly onto the 2D scenes. We evaluate on KITTI3D against other strong monocular methods and demonstrate that our approach doubles the AP on the 3D pose metrics on the official test set, defining the new state of the art.

Subfields: Aerospace Engineering, Computational Mechanics, Geology

View Paper: https://www.semanticscholar.org/paper/6338628d7499b3a438f92b01b05d7a621e80b29c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computational Mechanics

In Geology

Citations in the corpus

Delving into Localization Errors for Monocular 3D Object Detection (2021, methodology): “Specifically, [38, 26] use the CAD models as shape templates to get better object geometry.”
Disentangling Monocular 3D Object Detection (2019, methodology): “instance depth estimation predicts the central 3D depth of the nearest object instance, 3D location estimation seeks for the 3D bounding box center by exploiting 3D to 2D projections at given instance depth estimations, and local corner regression directly predicts the eight 3D bounding box corners ”
Learning Depth-Guided Convolutions for Monocular 3D Object Detection (2019, background): “[34] proposes a 3D synthetic data augmentation algorithm via in-painting recovered meshes directly onto the 2D scenes.”
FCOS3D: Fully Convolutional One-Stage Monocular 3D Object Detection (2021, unknown): “Some previous work [30, 20] keeps predicting 2D boxes and further regresses 3D attributes on top of 2D centers and regions of interest.”
MonoPair: Monocular 3D Object Detection Using Pairwise Spatial Relationships (2020, methodology): “An alternative family of methods is to predict a stand-alone depth or disparity information of the monocular image at the first stage [25, 26, 38, 41].”
3D Packing for Self-Supervised Monocular Depth Estimation (2019, background): “Depth from monocular camera configurations can provide useful cues for a wide array of tasks [23, 30, 34, 36], producing dense depth maps that could complement or eventually replace expensive range sensors.”
Categorical Depth Distribution Network for Monocular 3D Object Detection (2021, methodology): “Estimated depth maps can be used in combination with images to perform the 3D detection task [38, 64, 36, 13].”
Language-Grounded Indoor 3D Semantic Segmentation in the Wild (2022, background): “…towards data-sampling-based re-balancing, for instance random oversampling of underrepresented classes [3,54,50], sampling novel poses of known categories [32], undersampling overrepresented classes [35], frequency-based sampling [25], as well as feature-based or generative sampling [40,39,55].”
Geometry Uncertainty Projection Network for Monocular 3D Object Detection (2021, background): “The monocular 3D object detection aims to predict 3D bounding boxes from a single given image [13, 17, 21, 25, 31, 43].”
Objects are Different: Flexible Monocular 3D Object Detection (2021, unknown): “Considering the difficulty in perceiving 3D environments from 2D images, most existing methods for monocular 3D object detection utilize extra information to simplify the task, which includes pretrained depth estimation modules [30, 45, 46, 47], annotated keypoints [2] and CAD models [32].”
GDRNPP: A Geometry-Guided and Fully Learning-Based Object Pose Estimator (2021, unknown): “It has wide applicability to many real-world tasks such as robotic manipulation [1], [2], [3], augmented reality [4], [5] and autonomous driving [6], [7].”
Accurate Monocular 3D Object Detection via Color-Embedded 3D Reconstruction for Autonomous Driving (2019, methodology): “Simply using depth map as an additional channel of RGB image such as [39, 20], and then expecting neural network to extract effective features automatically is not the best solution.”
Is Pseudo-Lidar needed for Monocular 3D Object detection? (2021, background): “Hence, monocular 3D detection is an active research area [42, 57, 39, 69], owing to its potentially wide-ranging impact and the ubiquity of cameras.”
SMOKE: Single-Stage Monocular 3D Object Detection via Keypoint Estimation (2020, methodology): “Following prior work [22, 31], the orientation can be encoded as a 4D quaternion to formulate 3D bounding box.”
Cross-view Transformers for real-time Map-view Semantic Segmentation (2022, methodology): “ROI10D [25] lifts 2D detections into 3D using depth estimates then regresses 3D bounding boxes.”