Real-Time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor

By Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, Christian Theobalt. IEEE International Conference on Computer Vision, 2017

333 citations recorded by the source corpus.

We present an approach for real-time, robust and accurate hand pose estimation from moving egocentric RGB-D cameras in cluttered real environments. Existing methods typically fail for hand-object interactions in cluttered scenes imaged from egocentric viewpoints—common for virtual or augmented reality applications. Our approach uses two subsequently applied Convolutional Neural Networks (CNNs) to localize the hand and regress 3D joint locations. Hand localization is achieved by using a CNN to estimate the 2D position of the hand center in the input, even in the presence of clutter and occlusions. The localized hand position, together with the corresponding input depth value, is used to generate a normalized cropped image that is fed into a second CNN to regress relative 3D hand joint locations in real time. For added accuracy, robustness and temporal stability, we refine the pose estimates using a kinematic pose tracking energy. To train the CNNs, we introduce a new photorealistic dataset that uses a merged reality approach to capture and synthesize large amounts of annotated data of natural hand interaction in cluttered scenes. Through quantitative and qualitative evaluation, we show that our method is robust to self-occlusion and occlusions by objects, particularly in moving egocentric perspectives.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/9b2a27accc13bb958debd02142610cc37bbf6bf6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

3D Hand Shape and Pose Estimation From a Single RGB Image (2019, methodology): “However, existing hand RGB image datasets [43, 62, 63, 33] only provide the annotations of 2D/3D hand joint locations, and they do not contain any 3D hand shape annotations.”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, background): “Moreover, such datasets typically feature a limited number of objects [46, 57, 60, 61] while still requiring manual annotations [10] or post-processing [56, 68].”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, neutral): “However, existing ones [12, 26, 28, 55] cannot yet reach the ﬁdelity and realism to generalize to real datasets.”
Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation (2018, methodology): “In our work, we circumvent the object occlusion by working in a mixed reality scenario similar to [23].”
ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation (2022, methodology): “To create a personalized full body-and-hand template mesh for each subject, we obtain 3D scans of the subjects in varying poses using a 3dMD scanner [22].”
3D Hand Shape and Pose From Images in the Wild (2019, methodology): “To evaluate robustness to occlusions and clutter, we use the EGODEXTER dataset [28] that displays a hand from an egocentric view interacting with various objects.”
Visual Prompt Multi-Modal Tracking (2023, background): “For example, [35] assists with additional depth information for hand tracking in occlusion scenes.”
Hand Pose Estimation via Latent 2.5D Heatmap Regression (2018, methodology): “The EgoDexter (ED) dataset [62] consists of 4 test sequences (3190 frames) recorded with a body-mounted camera from egocentric viewpoints and contains cluttered backgrounds, fast camera motion, and complex interactions with various objects.”
Monocular Real-Time Hand Shape and Motion Capture Using Multi-Modal Data (2020, methodology): “[24] constructed a photorealistic dataset for better robustness.”
First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations (2017, background): “From a hand pose estimation perspective, [48] proposed a small synthetic dataset of static poses and thus could not succesfully train data-hungry algorithms, recently relieved by larger synthetic datasets [10, 35].”
WorkingHands: A Hand-Tool Assembly Dataset for Image Segmentation and Activity Mining (2019, methodology): “† : [23] contains only background segmentation and doesn’t separate hand from object.”
FreiHAND: A Dataset for Markerless Capture of Hand Pose and Shape From Single RGB Images (2019, methodology): “Building on the SynthHands [20] dataset Mueller et al.”
HOnnotate: A Method for 3D Annotation of Hand and Object Poses (2019, background): “Methods for 3D pose estimation of rigid objects and hands from monocular images have made significant progress recently, thanks to the development of Deep Learning, and the creation of large datasets or the use of synthetic images for training [32, 35, 46, 55, 71, 73].”
Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals (2017, background): “The field of 3D hand pose estimation has advanced rapidly, both in terms of accuracy [2, 4, 5, 6, 7, 26, 28, 32, 38, 45, 48, 49, 53, 54, 57, 60, 62] and dataset quality [10, 22, 43, 47, 52, 59].”
Whole-Body Human Pose Estimation in the Wild (2020, background): “Most approaches of hand pose estimation have to use labrecorded [56,13] or synthetic datasets [34,35,50], hampering the performance of the existing methods in real-world scenarios.”
HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation (2020, background): “Recently, researchers have been interested in using neural networks to solve the IK problem in robotic control [10], motion retargeting [66] and hand pose estimation [41, 22].”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, methodology): “Synthetic Synthetic data has been increasingly used for hand pose estimation [25, 49, 23, 14].”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, background): “While a significant amount of research has focused on predicting the pose of hands [27, 54, 56, 58, 70, 93, 94, 98] or objects [5, 48, 59, 78, 83, 90] in isolation, joint understanding of handobject interactions has received far less attention.”
Cross-Modal Deep Variational Hand Pose Estimation (2018, background): “Aiming at more mobile usage scenarios, recent work has proposed hybrid methods for hand-pose estimation from body-worn cameras under heavy occlusion [13].”
Sim2real transfer learning for 3D human pose estimation: motion to the rescue (2019, background): “Again, depth has proven useful [49, 76, 80], where state-of-the-art algorithms typically incorporate some form of generative model in-the-loop at test time.”