Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies

By Hanbyul Joo, Tomas Simon, Yaser Sheikh. 2018

564 citations recorded by the source corpus.

We present a unified deformation model for the markerless capture of human movement at multiple scales, including facial expressions, body motion, and hand gestures. An initial model is generated by locally stitching together models of the individual parts of the human body, which we refer to as "Frank". This model enables the full expression of part movements, including face and hands, by a single seamless model. We capture a dataset of people wearing everyday clothes and optimize the Frank model to create "Adam": a calibrated model that shares the same skeleton hierarchy as the initial model with a simpler parameterization. Finally, we demonstrate the use of these models for total motion tracking in a multiview setup, simultaneously capturing the large-scale body movements and the subtle face and hand motion of a social group of people.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, unknown): “…and performance capture Per-ceiving humans from monocular RGB data [12, 29, 31, 41, 43, 44, 58, 59, 64, 87] and under multiple views [37–40 , 62] Following the success of pixel-aligned implicit function learning [64, 65], recent methods can capture human performance from sparse [38,…”
Neural Radiance Flow for 4D View Synthesis and Video Processing (2020, unknown): “Therefore this paradigm is widely adopted for particular shape domains [24, 4, 12, 21, 27, 65, 80] such as human face [4], body [27], and hand [57].”
Monocular human pose estimation: A survey of deep learning-based methods (2020, methodology): “ferent body models (Joo et al., 2018).”
Machine Learning in Python: Main developments and technology trends in data science, machine learning, and artificial intelligence (2020, background): “While machine learning enables us to push the state-of-the-art in many fields such as natural language processing [116,123,163,164] and computer vision [131,165–167], certain applications involve sensitive data that demands responsible treatment.”
Whole-Body Human Pose Estimation in the Wild (2020, background): “Recent works [24,47,64] also explore the task of monocular 3D whole-body capture.”
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans (2020, methodology): “For the frame t , SMPL parameters S t are estimated from the multi-view images {I ct | c = 1 , ..., N c } using [25].”
Monocular Total Capture: Posing Face, Body, and Hands in the Wild (2018, methodology): “, body motion only with the SMPL model [30] or hand motion only by separate hand model of Frankenstein in [26]).”
GHUM & GHUML: Generative 3D Human Shape and Articulated Pose Models (2020, methodology): “A number of recent full body models like Adam, Frank, or SMPL-X[14, 31], combine legacy components for face, body and hands, but usually focus on constructing a consistent, joint parameterization with proper scaling on top of already learnt components, rather than on training a full body model, end-”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “Frankenstein & Adam: The Frankenstein model [30]”
ICON: Implicit Clothed humans Obtained from Normals (2021, background): “Mesh-based statistical body models [24, 32, 40, 45, 56] are a popular explicit representation for 3D human reconstruction.”
Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop (2019, background): “More recently, works have demonstrated fits for more expressive models in the multi-view [14], as well as the singleview setting [27, 41].”
Learning 3D Human Dynamics From Video (2018, background): “[30] fits a body model to a sequence of 3D point clouds and 3D joints obtained from multi-view stereo.”
Learning to Reconstruct People in Clothing From a Single RGB Camera (2019, methodology): “Top-down, model-based methods exploit a parametric body model consisting of pose and shape [7, 33, 48, 89, 57, 40] to regularize the fitting process.”
OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields (2018, methodology): “A growing number of computer vision and machine learning applications require 2D human pose estimation as an input for their systems [56], [57], [58], [59], [60], [61], [62].”
Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies (2021, methodology): “We use [22] to obtain the SMPL parameters from the 3D human poses and apply [15] to segment foreground humans.”
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild (2020, methodology): “As such, most methods employ statistical 3D body models with strong priors on shape learned from large-scale 3D scans and with known kinematic structure to model the articulation [5,69,43,49,29].”
On Self-Contact and Human Pose (2021, background): “Despite rapid progress in 3D human pose estimation [19, 20, 23, 31, 34, 40, 45], and despite the role that self-contact plays in our daily lives, only a handful of previous works discuss selfcontact.”