Expressive Body Capture: 3D Hands, Face, and Body From a Single Image
By Georgios Pavlakos, Vasileios Choutas, Nima Ghorbani, Timo Bolkart, Ahmed A. A. Osman, Dimitrios Tzionas, Michael J. Black. 2019
2,497 citations recorded by the source corpus.
To facilitate the analysis of human actions, interactions and emotions, we compute a 3D model of human body pose, hand pose, and facial expression from a single monocular image. To achieve this, we use thousands of 3D scans to train a new, unified, 3D model of the human body, SMPL-X, that extends SMPL with fully articulated hands and an expressive face. Learning to regress the parameters of SMPL-X directly from images is challenging without paired images and 3D ground truth. Consequently, we follow the approach of SMPLify, which estimates 2D features and then optimizes model parameters to fit the features. We improve on SMPLify in several significant ways: (1) we detect 2D features corresponding to the face, hands, and feet and fit the full SMPL-X model to these; (2) we train a new neural network pose prior using a large MoCap dataset; (3) we define a new interpenetration penalty that is both fast and accurate; (4) we automatically detect gender and the appropriate body models (male, female, or neutral); (5) our PyTorch implementation achieves a speedup of more than 8x over Chumpy. We use the new method, SMPLify-X, to fit SMPL-X to both controlled images and images in the wild. We evaluate 3D accuracy on a new curated dataset comprising 100 images with pseudo ground-truth. This is a step towards automatic expressive human capture from monocular RGB data. The models, code, and data are available for research purposes at https://smpl-x.is.tue.mpg.de.
Subfields: Computer Vision and Pattern Recognition, 3D Vision & Pose, Human-Computer Interaction
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jitendra Malik
- Dacheng Tao
- Ming-Hsuan Yang
- Trevor Darrell
- Xiaogang Wang
- Cordelia Schmid
- Andrea Vedaldi
In 3D Vision & Pose
- Michael J. Black
- Jitendra Malik
- Angjoo Kanazawa
- Gerard Pons-Moll
- Christian Theobalt
- Dimitrios Tzionas
- Kyoung Mu Lee
- Gyeongsik Moon
In Human-Computer Interaction
- Michael J. Black
- Jitendra Malik
- Dimitrios Tzionas
- Gyeongsik Moon
- Kyoung Mu Lee
- Angjoo Kanazawa
- Georgios Pavlakos
- Nasser Kehtarnavaz
Citations in the corpus
- Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
- End-to-End Human Pose and Mesh Reconstruction with Transformers (2020, methodology): “We also use the pseudo 3D mesh labels provided in [19], which are fitted with SMPLify-X [33].”
- Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop (2019, background): “Future work could consider extending this approach to capture multiple people [42, 43], or incorporate more expressive models of the human body [14, 26].”
- PARE: Part Attention Regressor for 3D Human Body Estimation (2021, methodology): “In estimating human shape and pose, many methods output the parameters of 3D human body models [3, 33, 39].”
- ContactPose: A Dataset of Grasps with Object Contact and Hand Pose (2020, methodology): “This formulation has also been used to penalize self-penetration and environment collision [43,61].”
- BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, methodology): “The last decade has seen rapid progress in modelling the appearance of humans ranging from body pose, shape [52, 58, 60, 61, 81], faces [74] and even detailed clothing [5, 7, 11, 57, 65].”
- HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation (2020, methodology): “This optimization paradigm was further extended with silhouette cues [26], volumetric grid [65], multiple people [73] and whole-body parametric model [47].”
- NeuMan: Neural Human Radiance Field from a Single Video (2022, methodology): “We hope to apply more expressive body models to model hand gestures [31], even garment dynamics [24], to mitigate this problem.”
- PointOdyssey: A Large-Scale Synthetic Dataset for Long-Term Point Tracking (2023, methodology): “We re-target the source motions represented as SMPL-X [44] sequences to target characters, using the motion retargeting algorithm from the Rokoko Toolkit [49].”
- On Self-Contact and Human Pose (2021, methodology): “To make reasoning about contact between body parts, the hands, and the face possible, we represent pose and shape with the SMPL-X [34] body model, which realistically captures the body surface details, including the hands and face.”
- Deep Learning-based Human Pose Estimation: A Survey (2020, background): “[183] introduced a new model, named SMPL-X, that can also predict fully articulated hands and facial landmarks.”
- 3D Morphable Face Models—Past, Present, and Future (2019, neutral): “Whilst faces and bodies are mostly analyzed separately, there exist first models that include faces and bodies jointly [Joo et al. 2018; Pavlakos et al. 2019].”
- Human Motion Diffusion Model (2022, background): “The human motion manifold can also be learned without labels, as shown by Holden et al. (2016), V-Poser (Pavlakos et al., 2019), and more recently the dedicated MoDi architecture (Raab et al., 2022).”
- MotionCLIP: Exposing Human Motion Generation to CLIP Space (2022, methodology): “Finally, we choose for each motion sequence a random frame to be rendered using the Blender software and the SMPL-X add-on [Pavlakos et al. 2019] (See Figure 4).”
- Humans in 4D: Reconstructing and Tracking Humans with Transformers (2023, methodology): “For example, the use of the SMPL model [45] creates certain limitations, and leveraging improved models would allow us to model hand pose and facial expressions [56], or even capture greater age variation, e.g ., infants [26] and kids [55, 70].”
- Deep learning tools for the measurement of animal behavior in neuroscience (2019, background): “Recently dense-pose representations, i.e. 3D point clouds or meshes (Figure 1c), have become a popular and elegant way to capture the soft-tissue and shape of bodies, which are highly important features for person identification, fashion (i.e. clothing sales), and in medicine (21, 42, 43).”
- D3D-HOI: Dynamic 3D Human-Object Interactions from Videos (2021, neutral): “Most works in this area focus on separately re-constructing the human mesh [1, 10, 11, 15, 16, 27] or the individual 3D object [4, 18, 22, 31].”