Depth-Based Hand Pose Estimation: Methods, Data, and Challenges

By James Steven Supancic III, Grégory Rogez, Yi Yang, Jamie Shotton, Deva Ramanan. 2015

245 citations recorded by the source corpus.

Hand pose estimation has matured rapidly in recent years. The introduction of commodity depth sensors and a multitude of practical applications have spurred new advances. We provide an extensive analysis of the state-of-the-art, focusing on hand pose estimation from a single depth frame. To do so, we have implemented a considerable number of systems, and have released software and evaluation code. We summarize important conclusions here: (1) Coarse pose estimation appears viable for scenes with isolated hands. However, high precision pose estimation [required for immersive virtual reality and cluttered scenes (where hands may be interacting with nearby objects and surfaces) remain a challenge. To spur further progress we introduce a challenging new dataset with diverse, cluttered scenes. (2) Many methods evaluate themselves with disparate criteria, making comparisons difficult. We define a consistent evaluation criteria, rigorously motivated by human experiments. (3) We introduce a simple nearest-neighbor baseline that outperforms most existing systems. This implies that most systems do not generalize beyond their training sets. This also reinforces the under-appreciated point that training data is as important as the model itself. We conclude with directions for future progress.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

Learning from Simulated and Unsupervised Images through Adversarial Training (2016, background): “Many efforts have explored using synthetic data for various prediction tasks, including gaze estimation [40], text detection and classification in RGB images [8, 14], font recognition [39], object detection [9, 24], hand pose estimation in depth images [35, 34], scene recognition in RGB-D [10], sema”
Disentangling Latent Hands for Image Synthesis and Pose Estimation (2018, background): “Vision-based hand pose estimation has progressed very rapidly in the past years [27, 38], driven in part by its potential for use in human-computer interaction applications.”
Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals (2017, unknown): “[44] compared 13 methods on a new dataset and concluded that deep models are well-suited to pose estimation [44].”
Hand Keypoint Detection in Single Images Using Multiview Bootstrapping (2017, background): “Discriminative and hybrid approaches to depth-based hand pose estimation rely heavily on synthetic data [23].”
Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, unknown): “Visual inputs from the environment are processed in our brain to recognize objects of interest (object recognition [16, 22, 26, 34, 85]), identify modes of interaction to achieve a certain function (affordance prediction [7, 20, 53, 72, 76]), and position our hand(s) in a way that enables that funct”
Spatial Attention Deep Net with Partial PSO for Hierarchical Hybrid Hand Pose Estimation (2016, unknown): “Many prior works have achieved good performance by different methods [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16].”
Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild (2020, background): “This is significantly more challenging than hand pose estimation when a depth sensor is also available [58, 47, 16, 38, 54].”
Deep Kinematic Pose Regression (2016, unknown): “In this work, we propose to directly incorporate the articulated object model into the deep neutral network learning, which is the dominant approach for object pose estimation nowadays, for hand [8,20,21,28,31,40] or human body [10,16,17,19,29,30,32,34,39].”