BodyNet: Volumetric Inference of 3D Human Body Shapes

By Gul Varol, Duygu Ceylan, Bryan C. Russell, Jimei Yang, Ersin Yumer, Ivan Laptev, Cordelia Schmid. European Conference on Computer Vision, 2018

456 citations recorded by the source corpus.

Human shape estimation is an important task for video editing , animation and fashion industry. Predicting 3D human body shape from natural images, however, is highly challenging due to factors such as variation in human bodies, clothing and viewpoint. Prior methods addressing this problem typically attempt to fit parametric body models with certain priors on pose and shape. In this work we argue for an alternative representation and propose BodyNet, a neural network for direct inference of volumetric body shape from a single image. BodyNet is an end-to-end trainable network that benefits from (i) a volumetric 3D loss, (ii) a multi-view re-projection loss, and (iii) intermediate supervision of 2D pose, 2D body part segmentation, and 3D pose. Each of them results in performance improvement as demonstrated by our experiments. To evaluate the method, we fit the SMPL model to our network output and show state-of-the-art results on the SURREAL and Unite the People datasets, outperforming recent approaches. Besides achieving state-of-the-art performance, our method also enables volumetric body-part segmentation.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/bbb4e98488d65cee2e0acc2ca27b12235e1dbef1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning 3D Human Dynamics From Video (2018, background): “explore predicting a voxel representation of human body [45].”
Monocular Total Capture: Posing Face, Body, and Hands in the Wild (2018, methodology): “Direct estimation methods predict 3D human pose directly from images, in the form of direct coordinate regression [46, 55, 56], voxel [42, 32, 66] or depth map [73].”
Learning to Dress 3D People in Generative Clothing (2019, unknown): “† Group 1: BodyNet [48], DeepHuman [54], SiCloPe [34], PIFu [44], MouldingHumans [12].”
Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild (2020, methodology): “Body Mesh Recovery Approaches based on regressing MANO parameters described in the previous subsection originate from works on body mesh recovery [3, 28, 22, 48, 42, 50, 41, 39, 21, 55, 25].”
End-to-End Human Pose and Mesh Reconstruction with Transformers (2020, background): “For example, researchers have explored to represent human body using a 3D mesh [20, 7], a volumetric space [47], or an occupancy field [41, 42].”
3D Hand Shape and Pose Estimation From a Single RGB Image (2019, methodology): “A more recent method [54] predicts a volumetric representation of human body.”
Monocular human pose estimation: A survey of deep learning-based methods (2020, background): “In addition to the evaluation metrics for 3D joint coordinates, there is another evaluation measurement Mean Per-vertex Error to report the results of 3D body shape which report the error between predicted and ground truth meshes (Varol et al., 2018; Pavlakos et al., 2018b).”
DeepHuman: 3D Human Reconstruction From a Single Image (2019, methodology): “7, we qualitatively compare our work against three state-of-the-art deep learning based approaches for single view 3D human reconstruction: HMR[27], BodyNet[52] and SiCloPe[36].”
HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation (2020, methodology): “This optimization paradigm was further extended with silhouette cues [26], volumetric grid [65], multiple people [73] and whole-body parametric model [47].”
ARCH: Animatable Reconstruction of Clothed Humans (2020, methodology): “) superior to the state of the art [32, 40, 44].”
Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop (2019, methodology): “[38] proposed an extension of SMPLify to fit SMPL on the regressed volumetric representation of their network.”
HuMoR: 3D Human Motion Model for Robust Pose Estimation (2021, methodology): “Much progress has been made on building methods to recover 3D joint locations [76, 67, 66] or parameterized 3D pose and shape ( i.e ., SMPL [59]) from observations [98].”
Learning to Reconstruct People in Clothing From a Single RGB Camera (2019, methodology): “Other recent works integrate the SMPL model, or a voxel representation [74], as a layer within a network architecture [41, 55, 53, 73].”
Image-Based 3D Object Reconstruction: State-of-the-Art and Trends in the Deep Learning Era (2019, methodology): “To exploit domain-speciﬁc knowledge, Varol et al. [122] introduce BodyNet, a volumetric approach for inferring, from a single RGB image, the 3D human body shape, along with its 2D and 3D pose, and its partwise segmentation.”
PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization (2019, methodology): “While voxel representations [59] can be applied in a fully-convolutional manner, the memory intensive nature of the representation inherently restrict its ability to produce fine-scale detailed surfaces.”
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, methodology): “Humans with voxels: From images, CNN based reconstruction of humans represented as voxels [70, 17, 82] or depth-maps [16, 62, 38] typically produce more details than mesh or template-based representations, because predictions are aligned with the input pixels.”
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild (2020, background): “Another line of work develops a learning-based framework, using a feed-forward model to directly predict the parameters of the body model from a single image [61,53,30,47,63,44].”
Single Image 3D Hand Reconstruction with Mesh Convolutions (2019, background): “[37] combine segmentation, 2D pose, and 3D pose predictions to infer the volumetric representation of the body into which they fit the”