3D Human Pose Estimation with Spatial and Temporal Transformers

By Ce Zheng, Sijie Zhu, Mat'ias Mendieta, Taojiannan Yang, Chen Chen, Zhengming Ding. IEEE International Conference on Computer Vision, 2021

692 citations recorded by the source corpus.

Transformer architectures have become the model of choice in natural language processing and are now being introduced into computer vision tasks such as image classification, object detection, and semantic segmentation. However, in the field of human pose estimation, convolutional architectures still remain dominant. In this work, we present PoseFormer, a purely transformer-based approach for 3D human pose estimation in videos without convolutional architectures involved. Inspired by recent developments in vision transformers, we design a spatial-temporal transformer structure to comprehensively model the human joint relations within each frame as well as the temporal correlations across frames, then output an accurate 3D human pose of the center frame. We quantitatively and qualitatively evaluate our method on two popular and standard benchmark datasets: Human3.6M and MPI-INF-3DHP. Extensive experiments show that PoseFormer achieves state-of-the-art performance on both datasets. Code is available at https://github.com/zczcwh/PoseFormer

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/7507603da9711c0e43e29f3094f33d9337e7dfbd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

MotionBERT: A Unified Perspective on Learning Human Motion Representations (2022, methodology): “(a) is conceptually similar to PoseFormer [127], MH-Former [52], and MixSTE [126].”
TokenPose: Learning Keypoint Tokens for Human Pose Estimation (2021, methodology): “PoseFormer [43] introduces Transformer for 3D pose estimation, based on 2D pose sequences in video frames.”
MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (2022, methodology): “In high-level vision, Transformer has been widely applied in image classification [7, 19, 25, 27, 49, 65], object detection [6, 22, 23, 60, 86, 93], semantic segmentation [16, 69,76,77,90,91], human pose estimation [14,15,32,37,42, 43, 53, 79, 89], etc.”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “[305] presented the irst purely transformer-based approach, named PoseFormer, for 3D HPE without convolutional architectures involved.”
MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation (2021, methodology): “6M following [2, 24, 34] and ground truth 2D pose for MPI-INF-3DHP following [4, 20, 47].”
MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video (2022, methodology): “Both [19] and [50] have to fix the order of spatial and temporal encoders, and only the central frame of video is reconstructed.”
SkySense: A Multi-Modal Remote Sensing Foundation Model Towards Universal Interpretation for Earth Observation Imagery (2023, background): “This de-sign delivers strong modeling of RSI sequences while us-ing substantially less parameters compared to common 3D structures [50, 87].”