GHUM & GHUML: Generative 3D Human Shape and Articulated Pose Models

By Hongyi Xu, Eduard Gabriel Bazavan, Andrei Zanfir, William T. Freeman, Rahul Sukthankar, Cristian Sminchisescu. Computer Vision and Pattern Recognition, 2020

404 citations recorded by the source corpus.

We present a statistical, articulated 3D human shape modeling pipeline, within a fully trainable, modular, deep learning framework. Given high-resolution complete 3D body scans of humans, captured in various poses, together with additional closeups of their head and facial expressions, as well as hand articulation, and given initial, artist designed, gender neutral rigged quad-meshes, we train all model parameters including non-linear shape spaces based on variational auto-encoders, pose-space deformation correctives, skeleton joint center predictors, and blend skinning functions, in a single consistent learning loop. The models are simultaneously trained with all the 3d dynamic scan data (over 60,000 diverse human configurations in our new dataset) in order to capture correlations and ensure consistency of various components. Models support facial expression analysis, as well as body (with detailed hand) shape and pose estimation. We provide fully train-able generic human models of different resolutions- the moderate-resolution GHUM consisting of 10,168 vertices and the low-resolution GHUML(ite) of 3,194 vertices–, run comparisons between them, analyze the impact of different components and illustrate their reconstruction from image data. The models will be available for research.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/aaec96d6e9a0a4877dde4382dc7889d47c074524

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “There are also other popular volumetric models such as DYNA [194], Stitched Puppet model [318], Frankenstein & Adam [94], and GHUM & GHUML(ite) [263].”
Action-Conditioned 3D Human Motion Synthesis with Transformer VAE (2021, methodology): “Consequently, without loss of generality, we employ the SMPL body model [43], which is a disentangled body representation (similar to recent models [50, 51, 54, 64]).”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, background): “The last decade has seen rapid progress in modelling the appearance of humans ranging from body pose, shape [52, 58, 60, 61, 81], faces [74] and even detailed clothing [5, 7, 11,57,65].”
CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation (2022, neutral): “Given a single RGB image, 3D human pose and shape estimation aims to reconstruct human body meshes with the help of statistic models [3,30,38,54].”
Generative Adversarial Networks for Face Generation: A Survey (2022, background): “Similarly, several GAN works [3, 11, 25, 36, 38, 103, 127, 127, 142, 149] proposed complex GAN architectures for facial Derives from binary cross-entropy between real and generated distributions; suffers from vanishing gradients and mode collapse Computes earth mover distance between real and…”
Nerfies: Deformable Neural Radiance Fields (2020, background): “Domain-Specific Modeling: Many reconstruction methods use domain-specific knowledge to model the shape and appearance of categories with limited topological variation, such as faces [5, 7, 9], human bodies [30, 58], and animals [12, 64].”
STAR: Sparse Trained Articulated Human Body Regressor (2020, methodology): “Contemporaneous with our work, GHUM [37] builds on SMPL and its Rodrigues pose representation but reduces the pose parameters (including face and hands) to a 32-dimensional latent code.”
ICON: Implicit Clothed humans Obtained from Normals (2021, background): “Mesh-based statistical body models [29,38,48,53,64] are a popular explicit representation for 3D human reconstruction.”
Human Pose Regression with Residual Log-likelihood Estimation (2021, background): “[65] propose new 3D human shape and articulated pose models with the kinematic prior based on normalizing flows.”
Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction (2020, background): “Alternative methods based on GHUM[54]”