User-Specific Hand Modeling from Monocular Depth Sequences

By Jonathan Taylor, Richard V. Stebbing, Varun Ramakrishna, Cem Keskin, Jamie Shotton, Shahram Izadi, Aaron Hertzmann, Andrew William Fitzgibbon. 2014

101 citations recorded by the source corpus.

This paper presents a method for acquiring dense nonrigid shape and deformation from a single monocular depth sensor. We focus on modeling the human hand, and assume that a single rough template model is available. We combine and extend existing work on model-based tracking, subdivision surface fitting, and mesh deformation to acquire detailed hand models from as few as 15 frames of depth data. We propose an objective that measures the error of fit between each sampled data point and a continuous model surface defined by a rigged control mesh, and uses as-rigid-as-possible (ARAP) regularizers to cleanly separate the model and template geometries. A key contribution is our use of a smooth model based on subdivision surfaces that allows simultaneous optimization over both correspondences and model parameters. This avoids the use of iterated closest point (ICP) algorithms which often lead to slow convergence. Automatic initialization is obtained using a regression forest trained to infer approximate correspondences. Experiments show that the resulting meshes model the user's hand shape more accurately than just adapting the shape parameters of the skeleton, and that the retargeted skeleton accurately models the user's articulations. We investigate the effect of various modeling choices, and show the benefits of using subdivision surfaces and ARAP regularization.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Learning Category-Specific Mesh Reconstruction from Image Collections (2018, methodology): “These techniques have since been applied to model human bodies [1,19], hands [27,17], and more recently on quadruped animals [40].”
Automatic differentiation in machine learning: a survey (2015, background): “, 1999), Gaussian mixture model fitting, and hand tracking (Taylor et al., 2014), and provide a comprehensive benchmark of various AD tools for the computation of derivatives in these tasks.”
Depth-Based Hand Pose Estimation: Methods, Data, and Challenges (2015, unknown): “Reprojection error: Following past work, we evaluate pose estimation as a regression task that predicts a set of 3D joint locations [14, 21, 25, 36, 37].”
3D Hand Shape and Pose Estimation From a Single RGB Image (2019, methodology): “The basic idea is to ﬁt a generative 3D hand model to the input depth image with iterative optimization [49, 30, 23, 20, 51, 41].”
Realtime and Robust Hand Tracking from Depth (2014, methodology): “Nevertheless, using optimized personal hand model [26] and sphere approximation technique [20] should further improve the accuracy.”
Accurate, Robust, and Flexible Real-time Hand Tracking (2015, background): “We believe that building an accurate personalized model of each user’s hand [29] can only improve the quality of tracking.”
Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation (2015, methodology): “Recent approaches relax the assumption of a fixed predefined shape model, allowing for online non-rigid shape deformation (Taylor et al, 2014) that enables better data fitting and user-specific adaptation.”
3D Hand Shape and Pose From Images in the Wild (2019, background): “In order to better capture the shape of the hand, [32] defines scaling terms to allow bone length to vary, while [54] pre-calibrates the shape to fit the hand of interest.”
Embodied hands (2017, background): “Taylor et al. [2014] personalize a template hand mesh by adapting its shape to fit the depth data of a user’s hand using an off-line calibration step for each user.”
Spatial Attention Deep Net with Partial PSO for Hierarchical Hybrid Hand Pose Estimation (2016, background): “Many prior works have achieved good performance by different methods [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16].”
Learning an efficient model of hand shape variation from depth images (2015, methodology): “We compare against a baseline approach based on the ‘personalization’ procedure detailed in [28].”