Convolutional Mesh Regression for Single-Image Human Shape Reconstruction

By Nikos Kolotouros, Georgios Pavlakos, Kostas Daniilidis. Computer Vision and Pattern Recognition, 2019

588 citations recorded by the source corpus.

This paper addresses the problem of 3D human pose and shape estimation from a single image. Previous approaches consider a parametric model of the human body, SMPL, and attempt to regress the model parameters that give rise to a mesh consistent with image evidence. This parameter regression has been a very challenging task, with model-based approaches underperforming compared to nonparametric solutions in terms of pose estimation. In our work, we propose to relax this heavy reliance on the model's parameter space. We still retain the topology of the SMPL template mesh, but instead of predicting model parameters, we directly regress the 3D location of the mesh vertices. This is a heavy task for a typical network, but our key insight is that the regression becomes significantly easier using a Graph-CNN. This architecture allows us to explicitly encode the template mesh structure within the network and leverage the spatial locality the mesh has to offer. Image-based features are attached to the mesh vertices and the Graph-CNN is responsible to process them on the mesh structure, while the regression target for each vertex is its 3D location. Having recovered the complete 3D geometry of the mesh, if we still require a specific model parametrization, this can be reliably regressed from the vertices locations. We demonstrate the flexibility and the effectiveness of our proposed graph-based mesh regression by attaching different types of features on the mesh vertices. In all cases, we outperform the comparable baselines relying on model parameter regression, while we also achieve state-of-the-art results among model-based pose estimation approaches.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9ce6d7373d307eb14f7ee76357e442f920be631a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Monocular, One-stage, Regression of Multiple 3D People (2020, background): “Generally, they first detect regions with people and then extract the bounding-box-level features, which are used to regress each single 3D human mesh [11, 20, 21, 22, 23, 24, 39, 47, 51, 60].”
Humans in 4D: Reconstructing and Tracking Humans with Transformers (2023, methodology): “GraphCMR [36] uses a graph neural network for the prediction, METRO [42] and FastMETRO [10] use a trans-former, while Mesh Graphormer [43] adopts a hybrid be-tween the two.”
Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies (2021, background): “Based on SMPL, some works [48, 24, 27, 21, 13] reconstruct an animated human mesh from sparse camera views.”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “[103] regressed the locations of the SMPL mesh vertices using a Graph-CNN architecture.”
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration (2020, methodology): “SPIN [26]), thanks to multiple augmentation tricks in training.”
Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose (2020, methodology): “able 8. The train sets include Human3.6M, COCO, MPII [3] , LSP [23], LSP-Extended [24], UP [33], and MPIINF-3DHP [40]. Each method is trained on a dierent subset of them. In the table, the errors of [26,30,31] decrease by a large margin compared to the errors in Table 5 and 6. Although it shows that”
TailorNet: Predicting Clothing in 3D as a Function of Human Pose, Shape and Garment Style (2020, methodology): “Our ﬁrst observation is that, for a ﬁxed style (garment instance) and body shape, predicting high frequency clothing deformations as a function of pose is possible – perhaps surprisingly, our experiments show that, for this task, a simple multi-layer perceptron model (MLP) performs as well as or bet”
Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop (2019, methodology): “To put our results into perspective, we also compare with four recent baselines ([3, 15, 16, 17]).”
Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild (2020, methodology): “Our work is similar to Kolotouros et al [26], who encode an image and concatenate its features with coordinates of a template mesh in a canonical pose.”
Mesh Graphormer (2021, methodology): “Among the previous studies, the Graph Convolutional Neural Network (GCNN) [17, 5, 20] is one of the most popular options as it is able to model the local interactions between neighboring vertices based on a given adjacency matrix [32, 5, 20, 38].”
Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, background): “Meshes for humans: Since the introduction of the (mesh-based) SMPL human model [41] there have been a growing number of papers leveraging it to reconstructing shape and pose from point clouds, depth data and images [28, 32, 33, 47, 69, 79].”
Geo-PIFu: Geometry and Pixel Aligned Implicit Functions for Single-view Human Reconstruction (2020, background): “For example, [13, 15, 16, 25, 26] estimate or fit the shape and pose coefficients of a SMPL model from the input image.”
Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction (2020, background): “y slow, and are susceptible to local-minima. In contrast, deep learning based models predict body model parameters in a feed-forward pass [16,43,39] and use bottom-up 2D features for self-supervision [21,23,24,32,50,60] during training. These approaches are limited by the shape space of SMPL, can no”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, methodology): “While we refer the reader to (Kolotouros et al. 2019b; Kolotouros et al. 2019; Kolotouros et al. 2019a; Kocabas et al. 2020) for a broader review, it is worth mentioning that related work includes optimization-based approaches, which fit a 3D mesh to 2D image keypoints (Bogo et al. 2016; Lassner et…”
3D Dynamic Scene Graphs: Actionable Spatial Perception with Places, Objects, and Humans (2020, methodology): “[45], which directly regresses the 3D location”
I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image (2020, methodology): “2 GraphCMR [24] 78.”