TransPose: Keypoint Localization via Transformer

By Sen Yang, Zhibin Quan, Mu Nie, Wankou Yang. IEEE International Conference on Computer Vision, 2020

383 citations recorded by the source corpus.

While CNN-based models have made remarkable progress on human pose estimation, what spatial dependencies they capture to localize keypoints remains unclear. In this work, we propose a model called Trans-Pose, which introduces Transformer for human pose estimation. The attention layers built in Transformer enable our model to capture long-range relationships efficiently and also can reveal what dependencies the predicted key-points rely on. To predict keypoint heatmaps, the last attention layer acts as an aggregator, which collects contributions from image clues and forms maximum positions of keypoints. Such a heatmap-based localization approach via Transformer conforms to the principle of Activation Maximization [19]. And the revealed dependencies are image-specific and fine-grained, which also can provide evidence of how the model handles special cases, e.g., occlusion. The experiments show that TransPose achieves 75.8 AP and 75.0 AP on COCO validation and test-dev sets, while being more lightweight and faster than mainstream CNN architectures. The TransPose model also transfers very well on MPII benchmark, achieving superior performance on the test set when fine-tuned with small training costs. Code and pre-trained models are publicly available1.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6c9710487ffe7161bf5a7b4e436ee6a76fc114cf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Luc Van Gool
3. Jing Zhang
4. Yunhe Wang
5. Vishal M. Patel
6. Chunjing Xu
7. Kai Han
8. Mubarak Shah

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Luc Van Gool
3. Radu Timofte
4. Ming-Hsuan Yang
5. Yunhe Wang
6. Jing Zhang
7. Vishal M. Patel
8. Kai Han

Citations in the corpus

• MST++: Multi-stage Spectral-wise Transformer for Efficient Spectral Reconstruction (2022, methodology): “In high-level vision, Transformer has been widely applied in image classification [7, 19, 25, 27, 49, 65], object detection [6, 22, 23, 60, 86, 93], semantic segmentation [16, 69,76,77,90,91], human pose estimation [14,15,32,37,42, 43, 53, 79, 89], etc.”
• RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose (2023, methodology): “ViTPose [61] leverages the state-of-art transformer backbones to boost pose estimation accuracy, while TransPose [62] integrate transformer encoders with CNNs to efficiently capture long-range spatial relationships.”
• Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “The early exploration [269] proposed a transformer-based model for 2D HPE named TransPose, which utilizes the attention layers to predict the heatmaps of the keypoints and learn the ine-grained evidence for HPE in occlusion scenarios.”
• ViTPose: Simple Vision Transformer Baselines for Human Pose Estimation (2022, methodology): “2 TransPose-H/A6 [46] HRNet-W48 18 309 256x192 1/4 75.”
• A Survey on Vision Transformer (2020, methodology): “Recently there has been growing interest in using transformer for high/mid-level computer vision tasks, such as object detection [19], [291], [10], [263], [166], lane detection [144], segmentation [235], [228], [285] and pose estimation [102], [103], [138], [253].”
• TransWeather: Transformer-based Restoration of Images Degraded by Adverse Weather Conditions (2021, methodology): “Transformers in low-level vision: Since the introduction of Vision Transformer (ViT) [9] for visual recognition, transformers have been widely adopted for various computer vision tasks [12, 31, 49, 60, 75].”
• Mask-guided Spectral-wise Transformer for Efficient Hyperspectral Image Reconstruction (2021, background): “66,66], segmentation [6,28,55,64,65], human pose estimation [23, 24, 56], etc.”