Video Based Reconstruction of 3D People Models

By Thiemo Alldieck, Marcus A. Magnor, Weipeng Xu, Christian Theobalt, Gerard Pons-Moll. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

550 citations recorded by the source corpus.

This paper describes a method to obtain accurate 3D body models and texture of arbitrary people from a single, monocular video in which a person is moving. Based on a parametric body model, we present a robust processing pipeline to infer 3D model shapes including clothed people with 4.5mm reconstruction accuracy. At the core of our approach is the transformation of dynamic body pose into a canonical frame of reference. Our main contribution is a method to transform the silhouette cones corresponding to dynamic human silhouettes to obtain a visual hull in a common reference frame. This enables efficient estimation of a consensus 3D shape, texture and implanted animation skeleton based on a large number of frames. Results on 4 different datasets demonstrate the effectiveness of our approach to produce accurate 3D models. Requiring only an RGB camera, our method enables everyone to create their own fully animatable digital double, e.g., for social VR applications or virtual try-on for online fashion shopping.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/223185c0dea34ed7e72df12df264674d24e824ed

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance (2024, background): “Its widespread adoption spans fields like human reconstruction [1, 13] and interaction with environments [12, 23].”
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video (2022, methodology): “13-(c): We conducted experiments on the PeopleSnapshot dataset [1], and used 64 samples per ray for quick evaluation.”
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, methodology): “Such implicit representation has been combined with the SMPL [41] body model to significantly increase tracking robustness and accuracy [78].”
Survey on Emotional Body Gesture Recognition (2018, background): “estimation from single monocular images [39], [40], [41], [42], [72], [73], [74], [75] and from videos [76], or multi-person pose estimation [77], [78], [79] and 3D pose estimation [80] are all”
DoubleFusion: Real-Time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor (2018, background): “[1] reconstruct detailed shape including clothing from a monocular RGB video but the approach is off-line.”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, unknown): “This enables real-time reconstructions from single images [8, 18], detailed texturing and displacement mapping [4, 6], and alleviates manual rigging [3].”
PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction (2020, methodology): “For qualitative evaluation on multi-image reconstruction, we use the open-source dataset from VideoAvatar [52] which contains 24 RGB sequences of different subjects turning 360 degree with a rough A-pose in front of a camera.”
Neural Unsigned Distance Fields for Implicit Function Learning (2020, methodology): “They are used for shape registration [26], model ﬁtting [70, 2], to speed up inference in part based models [25], and for extracting skeletons and medial axis [22, 7].”
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB (2017, background): “Some works split the problem in two: ﬁrst estimate 2D joints and then lift them to 3D [58, 54, 9, 66, 35, 69, 1, 52, 20, 32, 6, 26, 38, 57, 2], e.g ., by database matching, neural network regression, or ﬁtting the SMPL body model [30].”
Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans (2020, methodology): “Following [2], we report the qualitative results on the People-Snapshot dataset.”
Neural Body Fitting: Unifying Deep Learning and Model Based Human Pose and Shape Estimation (2018, background): “Recently, using monocular video and geometric reasoning, accurate and detailed 3D shape, including clothing is obtained [3, 2].”
Multi-Garment Net: Learning to Dress 3D People From Images (2019, background): “Similar to [3, 5], we also restrict our setting to the scenario where the person is turning around in front of the camera.”
TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style (2020, background): “One way to achieve realism is to capture real clothing on humans from images [6, 5, 3, 4, 19], dynamic scans [41, 39] or RGBD [52, 51] and re-target it to novel shapes, but this is limited to re-animating the same motion on a novel shape.”
Tex2Shape: Detailed Full Human Body Geometry From a Single Image (2019, methodology): “We qualitatively compare our work against four relevant methods for monocular human shape reconstruction on the PeopleSnapshot dataset [4].”
Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction (2020, methodology): “We do so by registering SMPL+D model [8,25] to IP-Net outputs: outer surface, inner surface and part correspondences.”
Learning to Dress 3D People in Generative Clothing (2019, methodology): “Another group of methods are based on SMPL [1, 2, 3, 4, 8, 55].”
Convolutional Mesh Regression for Single-Image Human Shape Reconstruction (2019, methodology): “[1] for training and apply our model on hold-out sequences.”
ICON: Implicit Clothed humans Obtained from Normals (2021, methodology): “Another advantage of mesh-based statistical models, is that texture information can be easily accumulated through multi-view images or image sequences [4, 8], due to their consistent mesh topology.”