Full DOF tracking of a hand interacting with an object by modeling occlusions and physical constraints

By I. Oikonomidis, Nikolaos Kyriazis, Antonis A. Argyros. Vision, 2011

330 citations recorded by the source corpus.

Due to occlusions, the estimation of the full pose of a human hand interacting with an object is much more challenging than pose recovery of a hand observed in isolation. In this work we formulate an optimization problem whose solution is the 26-DOF hand pose together with the pose and model parameters of the manipulated object. Optimization seeks for the joint hand-object model that (a) best explains the incompleteness of observations resulting from occlusions due to hand-object interaction and (b) is physically plausible in the sense that the hand does not share the same physical space with the object. The proposed method is the first that solves efficiently the continuous, full-DOF, joint hand-object tracking problem based solely on markerless multicamera input. Additionally, it is the first to demonstrate how hand-object interaction can be exploited as a context that facilitates hand pose estimation, instead of being considered as a complicating factor. Extensive quantitative and qualitative experiments with simulated and real world image sequences as well as a comparative evaluation with a state-of-the-art method for pose estimation of isolated hands, support the above findings.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/da179820c71dfcdcafefddd4b18dc3e1d299c23b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

A Survey on Human Motion Analysis from Depth Data (2013, background): “Recently, it has been proposed to track the manipulated object and the hand at the same time to constrain the search space using collision constraints [142].”
Real-Time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor (2017, neutral): “Meth-ods for capturing complex hand-object interactions and object scanning were proposed [20, 1, 13, 41, 40, 21].”
Learning Joint Reconstruction of Hands and Manipulated Objects (2019, methodology): “Recent work obtains promising results for 2D and 3D hand pose estimation from monocular RGB images using convolutional neural networks [9, 20, 37, 45, 60, 61, 80].”
Hands Deep in Deep Learning for Hand Pose Estimation (2015, background): “Accurate hand pose estimation is an important requirement for many Human Computer Interaction or Augmented Reality tasks, and has attracted lots of attention in the Computer Vision research community [10, 11, 14, 15, 17, 22, 23, 29].”
Challenges in multimodal gesture recognition (2016, methodology): “Model-based methods for hand pose estimation (Oikonomidis et al., 2011; de La Gorce et al., 2011; Oikonomidis et al., 2010; Rehg and Kanade, 1995) typically match visual observations to instances of a predefined hand model.”
Weakly-Supervised 3D Hand Pose Estimation from Monocular RGB Images (2018, background): “While multi-view approaches [21, 35] alleviate the occlusion problem and provide decent accuracy, they require sophisticated mesh models and optimization strategies that prohibit them from real-time tasks.”
Task-Oriented Hand Motion Retargeting for Dexterous Manipulation Imitation (2018, background): “In the hand pose estimation field, the interaction with (real) objects has also been studied [9,28,25,27,23,7,32].”
Training a Feedback Loop for Hand Pose Estimation (2015, methodology): “Accurate hand pose estimation is an important requirement for many Human Computer Interaction or Augmented Reality tasks [10], and has been steadily regaining ground as a focus of research interest in the past few years [13, 14, 19, 23, 26, 30, 31, 37], probably because of the emergence of 3D sensor”
Accurate, Robust, and Flexible Real-time Hand Tracking (2015, background): “Proofs of concept [19, 18] suggest that extending our approach to deal with these cases should not be too challenging, at least for the model fitter.”
GanHand: Predicting Human Grasp Affordances in Multi-Object Scenes (2020, background): “A broad line of work aims to estimate the pose of a hand manipulating an object, either from RGB-D inputs [26, 25, 62, 65, 66], video sequences [57, 4, 48, 68] or single images [34, 35].”
Embodied hands (2017, methodology): “2a) which enable fast evaluation of distances [Oikonomidis et al. 2011a, 2012, 2014; Qian et al. 2014; Rehg and Kanade 1994; Tagliasacchi et al. 2015].”
Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation (2015, methodology): “For comparison we use the software released for 2All annotated sequences are available at http://files. is.tue.mpg.de/dtzionas/hand-object-capture.html. tracking a single hand (Oikonomidis et al, 2011a), with the parameter setups used also in the other works.”
Two-hand Global 3D Pose Estimation using Monocular RGB (2020, background): “[17] demonstrated the tracking of both the hand and an interacting object in 3D with 8 surrounding fixed cameras.”
Resolving 3D Human Pose Ambiguities With 3D Scene Constraints (2019, methodology): “The 3D hand-object community has also explored similar physical constraints, such as [37, 47, 51, 56, 67, 68] to name a few.”
Articulated and Generalized Gaussian Kernel Correlation for Human Pose Estimation (2016, methodology): “This method was further extended in [35], [36].”
Constraining dense hand surface tracking with elasticity (2020, neutral): “Some hand tracking approaches tackle the problem of penetration avoidance by detecting collisions using a sparse set of proxy spheres [Oikonomidis et al. 2011], approximate proxy geometries [Tzionas et al. 2016], sphere-meshes [Tkach et al. 2017] and 3D Gaussians [Mueller et al. 2019].”
Survey on 3D Hand Gesture Recognition (2016, methodology): “[42] extended [26] by jointly estimating the 26-DoF hand pose and the model parameters of”
GRAB: A Dataset of Whole-Body Human Grasping of Objects (2020, neutral): “contact and physics, for body-scene [6, 22, 23, 57, 76], body-object [30, 32, 37], or hand-object [24, 45, 54, 55, 62, 68, 69, 72, 77, 78] scenarios.”