Realtime and Robust Hand Tracking from Depth

By Chen Qian, Xiao Sun, Yichen Wei, Xiaoou Tang. 2014

506 citations recorded by the source corpus.

We present a realtime hand tracking system using a depth sensor. It tracks a fully articulated hand under large viewpoints in realtime (25 FPS on a desktop without using a GPU) and with high accuracy (error below 10 mm). To our knowledge, it is the first system that achieves such robustness, accuracy, and speed simultaneously, as verified on challenging real data. Our system is made of several novel techniques. We model a hand simply using a number of spheres and define a fast cost function. Those are critical for realtime performance. We propose a hybrid method that combines gradient based and stochastic optimization methods to achieve fast convergence and good accuracy. We present new finger detection and hand initialization methods that greatly enhance the robustness of tracking.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Touch-less interactive augmented reality game on vision-based wearable device (2015, background): “, Kinect, Leap Motion) have already become popular, and some novel gesture recognition approaches have been developed on these devices [58, 75], but they have not provided ubiquitous interfaces for augmented reality computing applications on smartphones or novel projected glasses yet.”
TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 1 ModDrop (2018, unknown): “In parallel, tracking-based approaches are advancing quickly [25], [26].”
Multi-scale Deep Learning for Gesture Detection and Localization (2014, background): “In parallel, tracking-based approaches are advancing quickly [23, 24].”
Hand Pose Estimation through Weakly-Supervised Learning of a Rich Intermediate Representation (2015, methodology): “A hybrid method [21] for real-time hand tracking uses a simple hand model consisting of a number of spheres.”
Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation (2015, methodology): “Recently, Schmidt et al (2014) extended the popular signed distance function (SDF) representation to articulated objects, while Qian et al (2014) combine a gradient based ICP approach with PSO, showing the complementary nature of the two approaches.”
A hand pose tracking benchmark from stereo matching (2017, methodology): “To evaluate hand tracking from stereo sequences, we implement two hand pose tracking methods [3, 17].”
Deep Global Registration (2020, background): “A variety of applications, including 3D reconstruction, tracking, pose estimation, and object detection, invoke 3D registration as part of their operation [35, 5, 33].”
DART: Dense Articulated Real-Time Tracking (2014, methodology): “As the key focus of DART is the ability to generalize across a wide variety of models, we evaluate its performance in multiple domains.”
Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals (2017, background): “Manually labeled datasets such as [31, 41] contain just a few thousand examples, making them unsuitable for large-scale training.”
Knowledge As Priors: Cross-Modal Knowledge Generalization for Datasets Without Superior Knowledge (2020, background): “, depth maps [32, 45, 59], point clouds [13, 23], or stereo images [55].”
First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations (2017, background): “3D hand pose estimation: Mainly due to the recent availability of RGB-D sensors, the field has made significant progress in object-less third-person view [22, 25, 28, 37, 39, 41, 45, 53, 60, 80] and more modest advances in first-person view [10, 35, 38, 48].”
Cascaded hand pose regression (2015, background): “In addition to the high accuracy, our regression is also very fast (> 300 FPS on CPU, single thread) and would be influential for real applications.”
GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB (2017, background): “While there is a large body of existing works that consider marker-free image-based hand tracking or pose estimation, many of them require depth cameras [34, 47, 39, 44, 27, 7, 54] or multi-view setups [41, 1, 56].”
Articulated and Generalized Gaussian Kernel Correlation for Human Pose Estimation (2016, methodology): “We evaluate our articulated pose tracking algorithm on two depth benchmark datasets, i.e., [8] (body) and [19] (hand), which shows that the accuracy of pose estimation is competitive compared to the best results reported so far [6], [10], [20].”