DenseFusion: 6D Object Pose Estimation by Iterative Dense Fusion

By Chen Wang, Danfei Xu, Yuke Zhu, Roberto Mart'in-Mart'in, Cewu Lu, Li Fei-Fei, Silvio Savarese. Computer Vision and Pattern Recognition, 2019

1,170 citations recorded by the source corpus.

A key technical challenge in performing 6D object pose estimation from RGB-D image is to fully leverage the two complementary data sources. Prior works either extract information from the RGB image and depth separately or use costly post-processing steps, limiting their performances in highly cluttered scenes and real-time applications. In this work, we present DenseFusion, a generic framework for estimating 6D pose of a set of known objects from RGB-D images. DenseFusion is a heterogeneous architecture that processes the two data sources individually and uses a novel dense fusion network to extract pixel-wise dense feature embedding, from which the pose is estimated. Furthermore, we integrate an end-to-end iterative pose refinement procedure that further improves the pose estimation while achieving near real-time inference. Our experiments show that our method outperforms state-of-the-art approaches in two datasets, YCB-Video and LineMOD. We also deploy our proposed method to a real robot to grasp and manipulate objects based on the estimated pose.

Subfields: Aerospace Engineering, Biomedical Engineering, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/79ed40c04328d8e1600a3a31004b4efb843dbab1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Biomedical Engineering

In Control and Systems Engineering

Citations in the corpus

ImVoteNet: Boosting 3D Object Detection in Point Clouds With Image Votes (2020, background): “Recently more works focus on fusing 2D and 3D features earlier in the process such as Multi-modal Voxelnet [45], AVOD [16], multi-sensor [20] and 3D-SIS [11].”
iNeRF: Inverting Neural Radiance Fields for Pose Estimation (2020, methodology): “To remove the cluttered background in test images, we use the publicly available segmentation masks provided by DenseFusion [47].”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, methodology): “Namely, we compute a point cloud from each of the five depth images and merge them into a single point cloud by using camera pose information; then, we fit our object model to this point cloud, taking the prediction from DenseFusion as initialization.”
HybridPose: 6D Object Pose Estimation Under Hybrid Representations (2020, background): “Pose estimation via intermediate representation is sensitive to outliers in predictions, which are introduced by occlusion and cluttered backgrounds [37, 32, 40].”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, background): “State-of-the-art approaches for both 3D object pose [37, 20, 34, 27, 40, 26, 19] and 3D hand pose estimation [49, 23, 18, 2, 9, 14, 31] rely on deep learning and thus require large datasets with labeled hand or object poses for training.”
Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes (2021, background): “Model-based grasping circumvents reasoning about the physics of contact and grasp generation by pre-defining a set of grasps in the object frame and transform those grasps according to the 6-DoF object pose [1, 2, 3, 4] or detected keypoints of the objects [5, 6].”
Self-supervised 6D Object Pose Estimation for Robot Manipulation (2019, background): “Deep neural networks further boost the performance by learning more discriminative features and classifiers [1], [2], [8], [9], [19].”
Shape Prior Deformation for Categorical 6D Object Pose and Size Estimation (2020, methodology): “deformation eld D; and (4) estimates the correspondence matrix A. On the consideration that the depth and color are two dierent modalities, we follow the pixel-wise dense fusion approach proposed in [32] to eectively extract RGB-D features from the observation. For point cloud V, we use an embedding”
GDRNPP: A Geometry-Guided and Fully Learning-Based Object Pose Estimator (2021, positive): “Nonetheless, with the advent of deep learning, meth-odsbasedonneuralnetworksbecomedominantininstance-level objectposeestimation[16],[17],[18],[19],[20],[21].”
DeepIM: Deep Iterative Matching for 6D Pose Estimation (2018, background): “Similar to the trend in image-based matching, recent approaches Wang et al. (2019) propose to learn point features for registration, such as applying deep neural networks to point clouds (Qi et al. 2017).”
CosyPose: Consistent multi-view multi-object 6D pose estimation (2020, methodology): “Such a refinement step provides important performance improvements and is becoming common practice [8,11] as a final stage of the estimation process.”
PVN3D: A Deep Point-Wise 3D Keypoints Voting Network for 6DoF Pose Estimation (2019, methodology): “To handle scenes with multi objects, previous methods [50, 53, 39] utilize existing detection or semantic segmentation architecture to pre-process the image and obtain RoIs (regions of interest) containing only single objects.”
KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects (2019, methodology): “Two error measures that are standard in the literature [5, 28, 31] are Area Under the Curve (AUC) and percentage of 3D keypoint errors <2cm.”