Detailed, Accurate, Human Shape Estimation from Clothed 3D Scan Sequences

By Chao Zhang, Sergi Pujades, Michael J. Black, Gerard Pons-Moll. Computer Vision and Pattern Recognition, 2017

301 citations recorded by the source corpus.

We address the problem of estimating human pose and body shape from 3D scans over time. Reliable estimation of 3D body shape is necessary for many applications including virtual try-on, health monitoring, and avatar creation for virtual reality. Scanning bodies in minimal clothing, however, presents a practical barrier to these applications. We address this problem by estimating body shape under clothing from a sequence of 3D scans. Previous methods that have exploited body models produce smooth shapes lacking personalized details. We contribute a new approach to recover a personalized shape of the person. The estimated shape deviates from a parametric model to fit the 3D scans. We demonstrate the method using high quality 4D data as well as sequences of visual hulls extracted from multi-view images. We also make available BUFF, a new 4D dataset that enables quantitative evaluation (http://buff.is.tue.mpg.de). Our method outperforms the state of the art in both pose estimation and shape estimation, qualitatively and quantitatively.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/8de82ceeab990471ccc7d714fdd69e460c54c223

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies (2018, background): “Since these body models often assume minimum clothing for subjects, explicit modeling for clothing is needed to capture clothed subjects [63, 39].”
Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction (2020, methodology): “UM[54] 4 BL. Bhatnagar et al. also exist[57]. The most popular image features are 2D joints [12], or 2D joints and silhouettes [8,7,19]. Some work have focused on estimating body shape under clothing [61,10,55], or capturing body shape and clothing jointly from scans [35]. These approaches are typic”
Learning to Dress 3D People in Generative Clothing (2019, methodology): “Therefore, we first scan subjects in both clothed and minimally-clothed conditions, then use the SMPL model with free deformation [1, 53] to register the scans.”
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, background): “In the supplementary video, we show an additional result on single-view reconstruction on the BUFF dataset [80] from video (without retraining nor fine tuning the model).”
ARCH: Animatable Reconstruction of Clothed Humans (2020, methodology): “Our test dataset contains 64 scans from the RenderPeople dataset, 207 scans from the AXYZ dataset, 26 scans from the BUFF dataset [55], and 2D images from the DeepFashion [27] dataset, representing clothed people with a large variety of complex clothing.”
PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction (2020, methodology): “In addition, for quantitative comparison and evaluation we also use the BUFF dataset [62], which provides 26 4D human sequences with various clothes and over 9000 scans in total.”
Learning to Reconstruct People in Clothing From a Single RGB Camera (2019, methodology): “To quantify shape reconstruction accuracy, we adjust the pose of the estimation to match the ground truth, following [83, 9].”
Tex2Shape: Detailed Full Human Body Geometry From a Single Image (2019, methodology): “Body shape under clothing has been estimated without [59] and jointly with a separate clothing layer [38] from 3D scans and from RGB-D [45].”
ICON: Implicit Clothed humans Obtained from Normals (2021, methodology): “Twindom AGORA THuman BUFF CAPE [1] [53] [39] [63] [59] [35, 42]”
Recovering Accurate 3D Human Pose in the Wild Using IMUs and a Moving Camera (2018, methodology): “We also scanned subjects and non-rigidly fitted SMPL to obtain 3D models similar to [27, 46].”
ClothCap (2017, methodology): “We minimize an objective function that forces the shape to remain inside the cloth and tight to the visible parts of the multi-cloth alignment; for more details on the objective function we refer to (Zhang et al. 2017).”
Deep Volumetric Video From Very Sparse Multi-view Performance Capture (2018, methodology): “To reduce the computational cost for the inverse rendering problem in many of these approaches, Pons-Moll et al. [54] propose a multi-cloth 3D model to reconstruct both body and clothes from 4D scan sequences, estimating an unclothed body shape using [55, 56] and tracking the clothing over time.”
Multi-Garment Net: Learning to Dress 3D People From Images (2019, methodology): “Garment Shape Space In order to factor out pose deformations from garment shape, we “unpose” the j garment registrations Ggj ∈ Rmg×3, similar to [64, 47].”