Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation

By Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, Juergen Gall. 2015

327 citations recorded by the source corpus.

Hand motion capture is a popular research field, recently gaining more attention due to the ubiquity of RGB-D sensors. However, even most recent approaches focus on the case of a single isolated hand. In this work, we focus on hands that interact with other hands or objects and present a framework that successfully captures motion in such interaction scenarios for both rigid and articulated objects. Our framework combines a generative model with discriminatively trained salient points to achieve a low tracking error and with collision detection and physics simulation to achieve physically plausible estimates even in case of occlusions and missing visual data. Since all components are unified in a single objective function which is almost everywhere differentiable, it can be optimized with standard optimization techniques. Our approach works for monocular RGB-D sequences as well as setups with multiple synchronized RGB cameras. For a qualitative and quantitative evaluation, we captured 29 sequences with a large variety of interactions and up to 150 degrees of freedom.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In 3D Vision & Pose

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
Monocular Real-Time Hand Shape and Motion Capture Using Multi-Modal Data (2020, background): “Some works [35, 32, 40, 36, 43] additionally leveraged discriminative predictions for initialization and regularization.”
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration (2020, methodology): “Similarly, there have been noticeable achievements in 3D hand pose estimation from a depth input [34, 48, 45, 47, 53, 58] or a single RGB image [62, 9, 15, 8, 11, 61].”
InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image (2020, methodology): “[37] is the most similar dataset with ours in that it is constructed to analyze RGB interacting handfocused sequences.”
Expressive Body Capture: 3D Hands, Face, and Body From a Single Image (2019, methodology): “is redirected to [4, 28], but for the sake of completion we include the mathematical formulation also here.”
Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, methodology): “To estimate human grasps from visual inputs, existing methods train models on large-scale datasets [7, 9, 31, 38, 83].”
Monocular Total Capture: Posing Face, Body, and Hands in the Wild (2018, unknown): “Most of previous work is based on depth image as input [40, 54, 49, 52, 65, 71].”
Grasping Field: Learning Implicit Representations for Human Grasps (2020, methodology): “Reconstructing hand and object jointly has been studied with both RGB input and RGBD input [62, 71, 72, 73, 78, 83, 86, 87, 88, 89].”
ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation (2022, methodology): “To create a personalized full body-and-hand template mesh for each subject, we obtain 3D scans of the subjects in varying poses using a 3dMD scanner [22].”
Disentangling Latent Hands for Image Synthesis and Pose Estimation (2018, background): “recorded datasets are much smaller, with only tens of sequences [30, 39].”
Weakly-Supervised 3D Hand Pose Estimation from Monocular RGB Images (2018, background): “In general, they can be classified into generative approaches [20, 34, 16], discriminative approaches [13, 40, 6, 7, 5, 8] and hybrid approaches [25, 31, 30].”
Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies (2018, background): “…move [17, 21, 44, 19]; (2) facial motion is captured at close range, mostly frontal, and assuming little global head motion [5, 24, 6, 9, 51]; (3) ﬁnger motion is also captured at very close distances from hands, where the hand regions are dominant in the sensor measurements [36, 49, 42, 50].”
Hand Pose Estimation via Latent 2.5D Heatmap Regression (2018, methodology): “Z^r)2 C2: This results in two values for the unknown variable Z root, one in front of the camera and one behind the camera. We choose the solution in front of the camera Z^ root= 0:5( b+ p b2 4ac)=a: (7) Given the value of Z root, P^2:5D, and the intrinsic camera parameters K, the scale normalized 3”
Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, methodology): “Moreover, such datasets typically feature a limited number of objects [46, 57, 60, 61] while still requiring manual annotations [10] or post-processing [56, 68].”
GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB (2017, background): “Numerous follow-up works have been proposed to improve the generative component [44, 46, 51, 52], and the learning-based discriminator [58, 16, 45, 49, 55, 7, 37, 22, 61, 5, 4, 29].”
Embodied hands (2017, methodology): “2e) is more realistic and better fits image data [Ballan et al. 2012; Tzionas et al. 2016].”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, methodology): “Pioneer works in [1, 79, 80] investigate hand-hand and hand-object interactions relying on optimization frameworks which might be slow and difficult to tune.”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, unknown): “Given the challenges involved, hand-object interactions have been tackled in multiview or RGB-D camera setups [36, 46].”