Neural Body Fitting: Unifying Deep Learning and Model Based Human Pose and Shape Estimation

By Mohamed Omran, Christoph Lassner, Gerard Pons-Moll, Peter Gehler, Bernt Schiele. International Conference on 3D Vision, 2018

526 citations recorded by the source corpus.

Direct prediction of 3D body pose and shape parameters remains a challenge even for highly parameterized, deep learning models. The representation of the prediction space is difficult to map to from the plain 2D image space, perspective ambiguities make the loss function noisy and training data is scarce. In this paper, we propose a novel approach (Neural Body Fitting (NBF)) that integrates a statistical body model as a layer within a CNN leveraging both reliable bottom-up body part segmentation and robust top-down body model constraints. NBF is fully differentiable and can be trained end-to-end from both 2D and 3D annotations. In detailed experiments we analyze how the components of our model improve model performance and present a robust, easy to use, end-to-end trainable framework for 3D human pose estimation from single 2D images.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ed48b221b81bc43b4ded24c20bf16c571e682ba6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Tex2Shape: Detailed Full Human Body Geometry From a Single Image (2019, methodology): “In recent work, the SMPL [32] model has been integrated into network architectures [25, 36, 35, 48].”
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB (2017, unknown): “Some works integrate SMPL within the CNN to exploit 3D and 2D annotations in an end-to-end fashion [39, 25, 42].”
I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image (2020, background): “8 NBF [35] 59.”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, background): “…et al. 2017; Zanfir et al. 2018; Kolotouros et al. 2019; Yang and Carlone 2020), and learning-based methods, which infer the mesh directly from pixel information (Tan et al. 2017; Kanazawa et al. 2018; Omran et al. 2018; Pavlakos et al. 2018; Kolotouros et al. 2019b; Kolotouros et al. 2019).”
Mesh Graphormer (2021, background): “However, as discussed in the literature [20, 29, 5, 42], it remains difficult to estimate accurate coefficients from a single image, and researchers [4, 30, 18, 27] are trying to predict 3D posture, learning with more visual evidence, or adopting the dense correspondence maps [42, 9] to improve the ”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, background): “It also enables optimization within the bounds of the learned prior [14, 19, 28] and weak-supervision when integrated in a differentiable form [30] into neural training processes [5, 23, 26, 45, 48, 68].”
Learning to Reconstruct People in Clothing From a Single RGB Camera (2019, methodology): “Other recent works integrate the SMPL model, or a voxel representation [74], as a layer within a network architecture [41, 55, 53, 73].”
PaMIR: Parametric Model-Conditioned Implicit Representation for Image-Based Human Reconstruction (2020, background): “Instead of optimizing mesh and skeleton parameters, recent approaches proposed to train deep neural networks that directly regress the 3D shape and pose parameters from a single image [11], [12], [13], [14], [15].”
Resolving 3D Human Pose Ambiguities With 3D Scene Constraints (2019, methodology): “Recent methods based on deep learning, extend 3D human pose estimation to complex scenes [32, 42, 48, 50] but the 3D accuracy is limited.”
Learning to Dress 3D People in Generative Clothing (2019, background): “Most approaches [9, 17, 23, 26, 27, 30, 36, 38, 46] output 3D body meshes from images, but not clothing.”
Multi-Garment Net: Learning to Dress 3D People From Images (2019, background): “current statistical models can not represent clothing, most works [7, 26, 40, 68, 48, 32, 22, 67, 31, 50, 8, 33, 44, 46] are restricted to inferring body shape alone.”
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, background): “Meshes for humans: Since the introduction of the (mesh-based) SMPL human model [41] there have been a growing number of papers leveraging it to reconstructing shape and pose from point clouds, depth data and images [28, 32, 33, 47, 69, 79].”
VIBE: Video Inference for Human Body Pose and Shape Estimation (2019, methodology): “Tremendous progress has been made on estimating 3D human pose and shape from a single image [11, 21, 25, 29, 35, 36, 38, 45, 48].”
Single Image 3D Hand Reconstruction with Mesh Convolutions (2019, methodology): “Neural Body Fitting [29] is an end-to-end system that integrates the SMPL model within a CNN.”
Sim2real transfer learning for 3D human pose estimation: motion to the rescue (2019, background): “Similar to our work, state-of-the-art approaches often rely on parametric body models to incorporate strong priors on 3D poses [5, 7, 8, 22, 24, 34, 40, 43, 52, 57, 74].”