Exploiting Temporal Contexts With Strided Transformer for 3D Human Pose Estimation

By Wenhao Li, Hong Liu, Runwei Ding, Mengyuan Liu, Pichao Wang, Wenming Yang. IEEE transactions on multimedia, 2021

302 citations recorded by the source corpus.

Despite the great progress in 3D human pose estimation from videos, it is still an open problem to take full advantage of a redundant 2D pose sequence to learn representative representations for generating one 3D pose. To this end, we propose an improved Transformer-based architecture, called Strided Transformer, which simply and effectively lifts a long sequence of 2D joint locations to a single 3D pose. Specifically, a Vanilla Transformer Encoder (VTE) is adopted to model long-range dependencies of 2D pose sequences. To reduce the redundancy of the sequence, fully-connected layers in the feed-forward network of VTE are replaced with strided convolutions to progressively shrink the sequence length and aggregate information from local contexts. The modified VTE is termed as Strided Transformer Encoder (STE), which is built upon the outputs of VTE. STE not only effectively aggregates long-range information to a single-vector representation in a hierarchical global and local fashion, but also significantly reduces the computation cost. Furthermore, a full-to-single supervision scheme is designed at both full sequence and single target frame scales applied to the outputs of VTE and STE, respectively. This scheme imposes extra temporal smoothness constraints in conjunction with the single target frame supervision and hence helps produce smoother and more accurate 3D poses. The proposed Strided Transformer is evaluated on two challenging benchmark datasets, Human3.6 M and HumanEva-I, and achieves state-of-the-art results with fewer parameters. Code and models are available at https://github.com/Vegetebird/StridedTransformer-Pose3D.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/3063b14c89886d2dd2da18cc37d024178831034d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Junsong Yuan
2. Hong Liu
3. Pichao Wang
4. Luc Van Gool
5. Jun Liu
6. Mubarak Shah
7. G. Wang
8. Xiaogang Wang

In Human-Computer Interaction

1. Hong Liu
2. Pichao Wang
3. Junsong Yuan
4. Luc Van Gool
5. Taojiannan Yang
6. Ju Shen
7. Mubarak Shah
8. Ruixu Liu

Citations in the corpus

• MixSTE: Seq2seq Mixed Spatio-Temporal Encoder for 3D Human Pose Estimation in Video (2022, methodology): “MPJPE(mm) PoseFormer (T=81) [50] Anatomy-aware (T=81) [4] Anatomy-aware (T=243) [4] STE (T=243) [19] AM (T=243) [24] VideoPose3D (T=243) [32] Spatial Correlation of each frame Alternated Learning of S-T Correlation”
• Deep Learning-based Human Pose Estimation: A Survey (2020, background): “[125] proposed a strided transformer encoder (STE) to reduce the sequence redundancy and computation cost.”
• MHFormer: Multi-Hypothesis Transformer for 3D Human Pose Estimation (2021, background): “Recently, Transformer [37] equipped with the powerfully global self-attention mechanism has received increasingly research interest in the computer vision community [10,19,26,44].”