MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation

By Wenhao Li, Hong Liu, Hao Tang, Pichao Wang, Luc Van Gool. Computer Vision and Pattern Recognition, 2021

408 citations recorded by the source corpus.

Estimating 3D human poses from monocular videos is a challenging task due to depth ambiguity and self-occlusion. Most existing works attempt to solve both issues by exploiting spatial and temporal relationships. However, those works ignore the fact that it is an inverse problem where multiple feasible solutions (i.e., hypotheses) exist. To relieve this limitation, we propose a Multi-Hypothesis Transformer (MHFormer) that learns spatio-temporal representations of multiple plausible pose hypotheses. In order to effectively model multi-hypothesis dependencies and build strong relationships across hypothesis features, the task is decomposed into three stages: (i) Generate multiple initial hypothesis representations; (ii) Model self-hypothesis communication, merge multiple hypotheses into a single converged representation and then partition it into several diverged hypotheses; (iii) Learn cross-hypothesis communication and aggregate the multi-hypothesis features to synthesize the final 3D pose. Through the above processes, the final representation is enhanced and the synthesized pose is much more accurate. Extensive experiments show that MHFormer achieves state-of-the-art results on two challenging datasets: Human3.6M and MPI-INF-3DHP. Without bells and whistles, its performance surpasses the previous best result by a large margin of 3% on Human3.6M. Code and models are available at https://github.com/Vegetebird/MHFormer.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/c5a7ad1ac5c462113f5c12a19d46596944f9b418

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Chen Chen
2. Nasser Kehtarnavaz
3. Wayne Wu
4. Zhaoyang Liu
5. Taojiannan Yang
6. Sijie Zhu
7. Ce Zheng
8. Ju Shen

Citations in the corpus

• MotionBERT: A Unified Perspective on Learning Human Motion Representations (2022, methodology): “(a) is conceptually similar to PoseFormer [135], MHFormer [56], and MixSTE [134].”
• Deep Learning-based Human Pose Estimation: A Survey (2020, background): “[121] further designed a multi-hypothesis transformer to exploit spatial-temporal representations of multiple pose hypotheses.”