End-to-End Human Pose and Mesh Reconstruction with Transformers

By Kevin Qinghong Lin, Lijuan Wang, Zicheng Liu. 2020

790 citations recorded by the source corpus.

We present a new method, called MEsh TRansfOrmer (METRO), to reconstruct 3D human pose and mesh vertices from a single image. Our method uses a transformer encoder to jointly model vertex-vertex and vertex-joint interactions, and outputs 3D joint coordinates and mesh vertices simultaneously. Compared to existing techniques that regress pose and shape parameters, METRO does not rely on any parametric mesh models like SMPL, thus it can be easily extended to other objects such as hands. We further relax the mesh topology and allow the transformer self-attention mechanism to freely attend between any two vertices, making it possible to learn non-local relationships among mesh vertices and joints. With the proposed masked vertex modeling, our method is more robust and effective in handling challenging situations like partial occlusions. METRO generates new state-of-the-art results for human mesh reconstruction on the public Human3.6M and 3DPW datasets. Moreover, we demonstrate the generalizability of METRO to 3D hand reconstruction in the wild, outperforming existing state-of-the-art methods on FreiHAND dataset.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Multiscale Vision Transformers (2021, background): “An emerging thread of work aims at applying transformers to vision tasks such as object detection [5], semantic segmentation [121, 105], 3D reconstruction [77], pose estimation [113], generative modeling [17], image retrieval [30], medical image segmentation [16,103,117], point clouds [45], video in”
Humans in 4D: Reconstructing and Tracking Humans with Transformers (2023, methodology): “GraphCMR [35] uses a graph neural network for the prediction, METRO [41] and FastMETRO [9] use a transformer, while Mesh Graphormer [42] adopts a hybrid between the two.”
MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video (2022, background): “[21] presented a transformer framework for both human mesh recovery and pose estimation from a single image but ignored the temporal information in the video.”
CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation (2022, neutral): “They could be modelbased [16,21] or model-free [34,26], and most of them are top-down methods, except for one bottom-up [49].”
MotionBERT: A Unified Perspective on Learning Human Motion Representations (2022, methodology): “Note that most previous works [24, 41, 46, 48, 54, 58, 73, 114] use more datasets other than COCO [61] during training, such as LSP [39], MPI-INF3DHP [79], etc.”
NeuMan: Neural Human Radiance Field from a Single Video (2022, background): “This is, in fact, a non-trivial problem, as human body pose estimators [13,42,18] operate in their own camera systems with often near-orthographic camera models.”
Mesh Graphormer (2021, methodology): “Similar to [22], we also extract the 2048 -Dim image feature vector from the last hidden layer of the CNN and perform positional encoding using the 3D coordinates of each vertex and body joint in a human template mesh.”
PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop (2021, neutral): “In Table 3 of the main manuscript, the results are reported using mean AP, and variants of AP including AP 50 (AP at OKS = 0.50), AP 75 (AP at OKS = 0.75), AP M for persons with medium sizes, and AP L for persons with large sizes.”
Reconstructing Hands in 3D with Transformers (2023, methodology): “The baselines include METRO [33], Mesh Graphormer [34] and FrankMocap [49].”
Exploiting Temporal Contexts With Strided Transformer for 3D Human Pose Estimation (2021, methodology): “Unlike DETR [46], ViT [48], or METRO [50] that directly apply Transformer to images, we utilize a Transformer-based architecture to effectively map 2D keypoints to 3D poses.”
Deep Learning-based Human Pose Estimation: A Survey (2020, background): “proposed METRO [131] and MeshGraphormer [132] that combine CNNs with transformer networks to regress SMPL mesh vertices from a single image.”
3D Human Pose Estimation with Spatial and Temporal Transformers (2021, methodology): “Also, temporal consistency is neglected in METRO, which limits the robustness of its estimations.”
Multimodal Learning With Transformers: A Survey (2022, background): “language [27], sign language [28], [29], [30], point cloud [31], symbolic knowledge (graph) [32], [33], multimodal knowledge graph [34], sketch drawing [35], [36], [37], [38], 3D object/scene [39], [40], [41], document [42], [43], programming code [44] and Abstract Syntax Tree (AST) – a kind of grap”
TransPose: Keypoint Localization via Transformer (2020, methodology): “There are also works [26, 28, 33] applying Transformers to 3D pose estimation.”
MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation (2021, methodology): “For the pose or shape estimation task, METRO [21] uses a Transformer encoder to reconstruct 3D human vertices from a single image.”
Uncertainty-Guided Transformer Reasoning for Camouflaged Object Detection (2021, background): “Recent works have tried to apply transformers to various vision tasks, such as object detection [2, 5, 66], image recognition [8,47], multi-object tracking [39,48], semantic segmentation [64], and human pose and mesh reconstruction [30].”
FaceFormer: Speech-Driven 3D Facial Animation with Transformers (2021, methodology): “In computer graphics, transformers have been exploited for 3D point cloud representations and 3D mesh, such as Point Transformer [68], Point Cloud Transformer [26] and Mesh Transformer [38].”