Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop

By Nikos Kolotouros, Georgios Pavlakos, Michael J. Black, Kostas Daniilidis. IEEE International Conference on Computer Vision, 2019

1,173 citations recorded by the source corpus.

Model-based human pose estimation is currently approached through two different paradigms. Optimization-based methods fit a parametric body model to 2D observations in an iterative manner, leading to accurate image-model alignments, but are often slow and sensitive to the initialization. In contrast, regression-based methods, that use a deep network to directly estimate the model parameters from pixels, tend to provide reasonable, but not pixel accurate, results while requiring huge amounts of supervision. In this work, instead of investigating which approach is better, our key insight is that the two paradigms can form a strong collaboration. A reasonable, directly regressed estimate from the network can initialize the iterative optimization making the fitting faster and more accurate. Similarly, a pixel accurate fit from iterative optimization can act as strong supervision for the network. This is the core of our proposed approach SPIN (SMPL oPtimization IN the loop). The deep network initializes an iterative optimization routine that fits the body model to 2D joints within the training loop, and the fitted estimate is subsequently used to supervise the network. Our approach is self-improving by nature, since better network estimates can lead the optimization to better solutions, while more accurate optimization fits provide better supervision for the network. We demonstrate the effectiveness of our approach in different settings, where 3D ground truth is scarce, or not available, and we consistently outperform the state-of-the-art model-based pose estimation approaches by significant margins. The project website with videos, results, and code can be found at https://seas.upenn.edu/~nkolot/projects/spin.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0c2b90305dbeb11950ff71718cb3331048124d50

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Reconstructing Hands in 3D with Transformers (2023, methodology): “For our training losses, we follow best practices for parametric human and hand reconstruction [17, 27, 30, 49] and supervise our model with a combination of 2D and 3D losses.”
ICON: Implicit Clothed humans Obtained from Normals (2021, methodology): “There are now several methods that take a single image and regress a minimally clothed 3D human model [6,7,18,21,33,48].”
Perceiving 3D Human-Object Spatial Arrangements from a Single Image in the Wild (2020, background): “While recent work has also demonstrated impressive results in estimating 3D structures, particularly human bodies, the focus is often on bodies [30,37] and objects [11,17] imaged in isolation or in controlled lab conditions [23,56].”
End-to-End Human Pose and Mesh Reconstruction with Transformers (2020, methodology): “…and shape coefﬁcients directly from an input image, recent works further propose to leverage various human body priors such as human skeletons [26, 39] or segmentation maps [34], and explore different optimization strategies [24, 22, 51, 14] and temporal information [23] to improve reconstruction.”
HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation (2020, methodology): “Recent studies develop the model-based methods [7, 18, 24] to obtain these parameters from the monocular RGB input and produce 3D pose and shape of human bodies.”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “, [6, 11, 102, 116, 168, 288, 290, 314], because it is compatible with existing rendering engines.”
Neural Unsigned Distance Fields for Implicit Function Learning (2020, methodology): “Meshes are common to represent humans and have been used for estimating pose, shape [38, 40, 41, 52, 76, 86] and clothing [3, 2, 11] from images, but topologies and detail are restricted by an underlying model like SMPL [47] or SMPL + Garments meshes (ClothCap [57], TailorNet [54] and GarNet [30],…”
I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image (2020, methodology): “All the previous works [20, 23, 24, 40] used SMPL parameters obtained by applying Mosh [27] on the marker data of Human3.”
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, background): “Meshes for humans: Since the introduction of the (mesh-based) SMPL human model [41] there have been a growing number of papers leveraging it to reconstructing shape and pose from point clouds, depth data and images [28, 32, 33, 47, 69, 79].”
D3D-HOI: Dynamic 3D Human-Object Interactions from Videos (2021, neutral): “Most works in this area focus on separately re-constructing the human mesh [1, 10, 11, 15, 16, 27] or the individual 3D object [4, 18, 22, 31].”
Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction (2020, background): “For example, [13, 15, 16, 25, 26] estimate or fit the shape and pose coefficients of a SMPL model from the input image.”
VIBE: Video Inference for Human Body Pose and Shape Estimation (2019, methodology): “VIBE outperforms all state-of-the-art models including SPIN [36] on the challenging in-the-wild datasets (3DPW and MPI-INF-3DHP) and obtains comparable result on Human3.”
PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop (2021, methodology): “We provide more qualitative results and compare our PyMAF with the state-of-the-art approach SPIN [9].”
BEHAVE: Dataset and Method for Tracking Human Object Interactions (2022, neutral): “Human reconstruction and performance capture Per-ceiving humans from monocular RGB data [12, 29, 31, 41, 43, 44, 58, 59, 64, 87] and under multiple views [37–40 , 62] Following the success of pixel-aligned implicit function learning [64, 65], recent methods can capture human performance from sparse…”
CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation (2022, neutral): “Like most tasks in computer vision, there are two approaches to do this: top-down [16,23,20,21] and bottom-up [49,56].”
Deep 3D human pose estimation: A review (2021, positive): “More recently, Pavlakos et al. (2019) introduce a strategy to utilize appearance consistency in a video with different views, which is helpful for model-based pose estimation.”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, methodology): “Camera intrinsics, bone lengths for setting a m , and pose θ k are initialized with [26] for every frame k .”
Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction (2020, neutral): “In contrast, deep learning based models predict body model parameters in a feed-forward pass [16,43,39] and use bottom-up 2D features for self-supervision [21,23,24,32,50,60] during training.”
Mesh Graphormer (2021, methodology): “We evaluate our model using the 3D pose accuracy, and report the accuracy using PA-MPJPE metric [43, 14, 19, 20].”