Two-hand Global 3D Pose Estimation using Monocular RGB

By Fanqing Lin, Connor Wilhelm, Tony R. Martinez. 2020

50 citations recorded by the source corpus.

We tackle the challenging task of estimating global 3D joint locations for both hands via only monocular RGB input images. We propose a novel multi-stage convolutional neural network based pipeline that accurately segments and locates the hands despite occlusion between two hands and complex background noise and estimates the 2D and 3D canonical joint locations without any depth information. Global joint locations with respect to the camera origin are computed using the hand pose estimations and the actual length of the key bone with a novel projection algorithm. To train the CNNs for this new task, we introduce a large-scale synthetic 3D hand pose dataset. We demonstrate that our system outperforms previous works on 3D canonical hand pose estimation benchmark datasets with RGB-only information. Additionally, we present the first work that achieves accurate global 3D hand tracking on both hands using RGB-only inputs and provide extensive quantitative and qualitative evaluation.

Subfields: Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism, 3D Vision & Pose, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Simon Stent
2. Tony R. Martinez
3. Fanqing Lin
4. Shenghao Zhou
5. Lingzhi Zhang
6. Jianbo Shi
7. Jitendra Malik
8. Luc Van Gool

In Endocrinology, Diabetes and Metabolism

1. Tony R. Martinez
2. Fanqing Lin

In 3D Vision & Pose

1. Tony R. Martinez
2. Fanqing Lin
3. Lingzhi Zhang
4. Shenghao Zhou
5. Simon Stent
6. Jianbo Shi

In Human-Computer Interaction

1. Tony R. Martinez
2. Fanqing Lin
3. Lingzhi Zhang
4. Shenghao Zhou
5. Simon Stent
6. Jianbo Shi

Citations in the corpus

• Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “More recently, many works [1,73,4,34,36,37,58,59,61,24,50] have applied deep networks for hand or object segmentation.”
• Ego2Hands: A Dataset for Egocentric Two-hand Segmentation and Detection (2020, methodology): “Finally, we jointly train the proposed HandSegNet [16] on both Ego3DHandsR and Ego2Hands using the following loss: Lcombined = Lsynthseg + Lrealseg + Lsynthenergy, (2) where Lsynth and Lreal are losses computed using synthetic and real-world data respectively.”