ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation

By Yufei Xu, Jing Zhang, Qiming Zhang, Dacheng Tao. Neural Information Processing Systems, 2022

997 citations recorded by the source corpus.

Although no specific domain knowledge is considered in the design, plain vision transformers have shown excellent performance in visual recognition tasks. However, little effort has been made to reveal the potential of such simple structures for pose estimation tasks. In this paper, we show the surprisingly good capabilities of plain vision transformers for pose estimation from various aspects, namely simplicity in model structure, scalability in model size, flexibility in training paradigm, and transferability of knowledge between models, through a simple baseline model called ViTPose. Specifically, ViTPose employs plain and non-hierarchical vision transformers as backbones to extract features for a given person instance and a lightweight decoder for pose estimation. It can be scaled up from 100M to 1B parameters by taking the advantages of the scalable model capacity and high parallelism of transformers, setting a new Pareto front between throughput and performance. Besides, ViTPose is very flexible regarding the attention type, input resolution, pre-training and finetuning strategy, as well as dealing with multiple pose tasks. We also empirically demonstrate that the knowledge of large ViTPose models can be easily transferred to small ones via a simple knowledge token. Experimental results show that our basic ViTPose model outperforms representative methods on the challenging MS COCO Keypoint Detection benchmark, while the largest model sets a new state-of-the-art. The code and models are available at https://github.com/ViTAE-Transformer/ViTPose.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/fdcad86866ca22d8417599deb41b54fe01487ce8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose (2023, methodology): “Compared with heatmap-based algorithms [24, 59, 61, 65], the SimCC-based algorithm achieves competitive accuracy with lower computational effort.”
ViTAEv2: Vision Transformer Advanced by Exploring Inductive Bias for Image Recognition and Beyond (2022, background): “Recently, it has been shown that the isotropic ViTAE can also achieve superior performance in human pose estimation (Xu et al., 2022).”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, background): “2) Universal Sequential SSL Models for Image Processing and CV: Contrastive predictive coding (CPC) [58] operates on the fundamental concept of acquiring informative representations through latent space predictions of future data using robust autoregressive models.”
Reconstructing Hands in 3D with Transformers (2023, methodology): “For our network, we use a large-scale transformer architecture [14, 57] which can successfully consume data of this scale.”
Advancing Plain Vision Transformer Toward Remote Sensing Foundation Model (2022, methodology): “Similarly, ViTPose [14] shows the potential of the plain structure on pose estimation tasks with a simpler”
Humans in 4D: Reconstructing and Tracking Humans with Transformers (2023, methodology): “Given any image, we first use an off-the-shelf detector [39] and a body keypoints estimator [78] to get bounding boxes and corresponding 2D keypoints.”