Learning to Reconstruct People in Clothing From a Single RGB Camera

By Thiemo Alldieck, Marcus A. Magnor, Bharat Lal Bhatnagar, Christian Theobalt, Gerard Pons-Moll. Computer Vision and Pattern Recognition, 2019

356 citations recorded by the source corpus.

We present Octopus, a learning-based model to infer the personalized 3D shape of people from a few frames (1-8) of a monocular video in which the person is moving with a reconstruction accuracy of 4 to 5mm, while being orders of magnitude faster than previous methods. From semantic segmentation images, our Octopus model reconstructs a 3D shape, including the parameters of SMPL plus clothing and hair in 10 seconds or less. The model achieves fast and accurate predictions based on two key design choices. First, by predicting shape in a canonical T-pose space, the network learns to encode the images of the person into pose-invariant latent codes, where the information is fused. Second, based on the observation that feed-forward predictions are fast but do not always align with the input images, we predict using both, bottom-up and top-down streams (one per view) allowing information to flow in both directions. Learning relies only on synthetic 3D data. Once learned, Octopus can take a variable number of frames as input, and is able to reconstruct shapes even from a single image with an accuracy of 5mm. Results on 3 different datasets demonstrate the efficacy and accuracy of our approach.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e4840bb6d97bb2464288a483e4374ef2a06b1822

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, methodology): “Such implicit representation has been combined with the SMPL [41] body model to significantly increase tracking robustness and accuracy [78].”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, methodology): “We use FrankMocap [63] to initialize SMPL’s pose from the images and then use instance speciﬁc optimization [6] to ﬁt the SMPL model to the segmented human point cloud.”
PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction (2020, methodology): “In contrast, the method in [53] is a learning-based method that deforms the SMPL template based on only 8 views of images.”
Multi-Garment Net: Learning to Dress 3D People From Images (2019, methodology): “• 3D vertex loss in posed space: L3DP = F−1∑ f=0 ||C(β,θf ,D)− C(β̂, θ̂f , D̂)||2 (16) • 2D segmentation loss: Unlike [3] we do not optimize silhouette overlap, instead we jointly optimize the projected per-garment segmentation against the input segmentation mask.”
Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction (2020, methodology): “l for inner and outer surfaces is better. 4.7 Using IP-Net Correspondences to Register Scans A very powerful use case for IP-Net is scan registration. Current state-of-the art registration approaches [6,25] for registering SMPL+D to 3D scans are tedious and cumbersome (as described in Sec. 3.1). We ”
TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style (2020, background): “One way to achieve realism is to capture real clothing on humans from images [6, 5, 3, 4, 19], dynamic scans [41, 39] or RGBD [52, 51] and re-target it to novel shapes, but this is limited to re-animating the same motion on a novel shape.”
ARCH: Animatable Reconstruction of Clothed Humans (2020, background): “There are multiple attempts to solve this task with video inputs [1, 2, 34, 38, 51], RGB-D data [52, 55] and in multi-view settings [3, 10, 11, 44, 45, 46, 47].”
Learning to Dress 3D People in Generative Clothing (2019, positive): “Therefore, we first scan subjects in both clothed and minimally-clothed conditions, then use the SMPL model with free deformation [1, 53] to register the scans.”
Tex2Shape: Detailed Full Human Body Geometry From a Single Image (2019, methodology): “Other recent works [4, 2] recover static body shape, and clothing as displacements on top of the SMPL body model [32] (model-based), or use a voxel representation [50, 33].”
Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies (2021, methodology): “To overcome this problem, [3, 2, 4] apply vertex displacements to the SMPL model to capture the human clothing and hair.”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, background): “SMPL has the advantage of encoding in a disentangled manner both the shape, the pose, and the pose speciﬁc details, and thus it has been extensively used in deep learning-based human body shape reconstruction [110], [116], [117], [118].”
PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization (2020, methodology): “Alternative methods consider additional free-form deformation on top of a parametric model space [1], and there exist also multiple approaches that predict depth maps of the target people as output [40, 11, 39].”
DeepHuman: 3D Human Reconstruction From a Single Image (2019, methodology): “To reconstruct a human body using a single-view RGB camera, methods have been proposed for rotating the camera while the target remains as static as possible [65], or keeping the camera static while the target rotates [3, 2, 1].”
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization (2019, methodology): “Inference techniques based on global representations [19, 30, 1] are more memory efficient, but cannot guarantee that details of input images are preserved.”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, methodology): “It also enables optimization within the bounds of the learned prior [14, 19, 28] and weak-supervision when integrated in a differentiable form [30] into neural training processes [5, 23, 26, 45, 48, 68].”