Embodied hands

By Javier Romero, Dimitrios Tzionas, Michael J. Black. ACM Transactions on Graphics, 2017

521 citations recorded by the source corpus.

Humans move their hands and bodies together to communicate and solve tasks. Capturing and replicating such coordinated activity is critical for virtual characters that behave realistically. Surprisingly, most methods treat the 3D modeling and tracking of bodies and hands separately. Here we formulate a model of hands and bodies interacting together and fit it to full-body 4D sequences. When scanning or capturing the full body in 3D, hands are small and often partially occluded, making their shape and pose hard to recover. To cope with low-resolution, occlusion, and noise, we develop a new model called MANO (hand Model with Articulated and Non-rigid defOrmations). MANO is learned from around 1000 high-resolution 3D scans of hands of 31 subjects in a wide variety of hand poses. The model is realistic, low-dimensional, captures non-rigid shape changes with pose, is compatible with standard graphics packages, and can fit any human hand. MANO provides a compact mapping from hand poses to pose blend shape corrections and a linear manifold of pose synergies. We attach MANO to a standard parameterized 3D body shape model (SMPL), resulting in a fully articulated body and hand model (SMPL+H). We illustrate SMPL+H by fitting complex, natural, activities of subjects captured with a 4D scanner. The fitting is fully automatic and results in full body models that move naturally with detailed hand motions and a realism not seen before in full body performance capture. The models and data are freely available for research purposes at http://mano.is.tue.mpg.de.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/fab5008ca5ca2d328395c1138885618fc209904d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

HOnnotate: A Method for 3D Annotation of Hand and Object Poses (2019, methodology): “We rely on the MANO hand model [50], and the 3D model of the objects.”
TEMOS: Generating diverse human motions from textual descriptions (2022, methodology): “i) the SMPL local rotations [8] (parent-relative joint rotation according to the kinematic tree) in the 6D continuous [18] representation (we load the AMASS data processed with SMPL-H [13], but we remove all rotations on the hands, resulting in 21 rotations) (21x6=126-dimensional features), ii) the ”
Two-hand Global 3D Pose Estimation using Monocular RGB (2020, methodology): “Some methods [5, 32, 9] estimated the low-dimensional parameters for a 3D deformable hand model [19] to fit the RGB inputs in order to retrieve the 3D canonical hand poses.”
HuMoR: 3D Human Motion Model for Robust Pose Estimation (2021, methodology): “For all experiments, we optimize using the neutral SMPL+H [31] body model in 3 stages.”
Grasp'D: Differentiable Contact-rich Grasp Synthesis for Multi-fingered Hands (2022, methodology): “[55] formulate a differentiable force closure metric and use gradient-based optimization to synthesize grasps with the MANO [75] hand model.”
GHUM & GHUML: Generative 3D Human Shape and Articulated Pose Models (2020, methodology): “In turn, SMPL-X attaches the FLAME[21] head to the SMPLH (body and hand) model[35] and refits it to an additional set of 5,586 scans.”
Constraining dense hand surface tracking with elasticity (2020, methodology): “Most existing approaches utilize generic skeleton-based models, e.g. , linear blend skinning (LBS) approaches [Taylor et al. 2016; Tan et al. 2016; Taylor et al. 2017] and MANO-based approaches [Romero et al. 2017; Mueller et al. 2019; Hasson et al. 2019; Baek et al. 2019].”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, methodology): “To represent 3D hand pose, we use the popular MANO hand model [28].”
Reconstructing Hand-Object Interactions in the Wild (2020, methodology): “We represent the hand using a parametric model defined by MANO [46]:”
Whole-Body Human Pose Estimation in the Wild (2020, background): “proposes a generative 3D model [47] to express body and hands.”