Exploiting Spatial-Temporal Relationships for 3D Pose Estimation via Graph Convolutional Networks

By Yujun Cai, Liuhao Ge, Jun Liu, Jianfei Cai, Tat-Jen Cham, Junsong Yuan, Nadia Magnenat-Thalmann. IEEE International Conference on Computer Vision, 2019

520 citations recorded by the source corpus.

Despite great progress in 3D pose estimation from single-view images or videos, it remains a challenging task due to the substantial depth ambiguity and severe self-occlusions. Motivated by the effectiveness of incorporating spatial dependencies and temporal consistencies to alleviate these issues, we propose a novel graph-based method to tackle the problem of 3D human body and 3D hand pose estimation from a short sequence of 2D joint detections. Particularly, domain knowledge about the human hand (body) configurations is explicitly incorporated into the graph convolutional operations to meet the specific demand of the 3D pose estimation. Furthermore, we introduce a local-to-global network architecture, which is capable of learning multi-scale features for the graph-based representations. We evaluate the proposed method on challenging benchmark datasets for both 3D hand pose estimation and 3D body pose estimation. Experimental results show that our method achieves state-of-the-art performance on both tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/50167a9e6bdc1f64a58fe8a3c990105fec0bd2bc

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, background): “For hands, [7] proposes a graph convolutional approach to learn temporal dependencies.”
3D Human Pose Estimation with Spatial and Temporal Transformers (2021, background): “However, instead of estimating 3D human pose from monocular images, videos can provide temporal information to improve accuracy and robustness [49, 10, 32, 8, 2, 44, 38].”
Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild (2020, background): “[10] proposed to build a spatiotemporal graph and use graph convolutions in the spectral domain to estimate a 3D pose.”
Deep Learning-based Human Pose Estimation: A Survey (2020, background): “Instead of estimating 3D human pose from monocular images, videos can provide temporal information to improve accuracy and robustness of 3D HPE, e.g., References [13, 34, 44, 187, 227, 248, 311, 312].”
Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose (2020, background): “AP defines the edge connections between the joints based on the human skeleton and symmetrical relationships [9], where (AP)ij = 1 if joints i and j are the same or connected, and (AP)ij = 0 otherwise.”
MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video (2022, methodology): “Besides, GCN [17] was also applied to the task by [1] to learn multi-scale features of human and hand poses.”
Deep 3D human pose estimation: A review (2021, unknown): “Nowadays, Cai et al. (2019) represent a 2D pose sequence as a graph and design a local-to-global network to estimate the corresponding 3D pose sequence, where the network can capture multi-scale features and learn a temporal constraint for the pose sequence.”
MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation (2021, methodology): “6M following [2, 24, 34] and ground truth 2D pose for MPI-INF-3DHP following [4, 20, 47].”