Effective Whole-body Pose Estimation with Two-stages Distillation
By Zhendong Yang, Ailing Zeng, Chun Yuan, Yu Li. 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), 2023
405 citations recorded by the source corpus.
Whole-body pose estimation localizes the human body, hand, face, and foot keypoints in an image. This task is challenging due to multi-scale body parts, fine-grained localization for low-resolution regions, and data scarcity. Meanwhile, applying a highly efficient and accurate pose estimator to widely human-centric understanding and generation tasks is urgent. In this work, we present a two-stage pose Distillation for Whole-body Pose estimators, named DWPose, to improve their effectiveness and efficiency. The first-stage distillation designs a weight-decay strategy while utilizing a teacher’s intermediate feature and final logits with both visible and invisible keypoints to supervise the student from scratch. The second stage distills the student model itself to further improve performance. Different from the previous self-knowledge distillation, this stage finetunes the student’s head with only 20% training time as a plug-and-play training strategy. For data limitations, we explore the UBody dataset that contains diverse facial expressions and hand gestures for real-life applications. Comprehensive experiments show the superiority of our proposed simple yet effective methods. We achieve new state-of-the-art performance on COCO-WholeBody, significantly boosting the whole-body AP of RTMPose-l from 64.8% to 66.5%, even surpassing RTMPose-x teacher with 65.3% AP. We release a series of models with different sizes, from tiny to large, for satisfying various downstream tasks. Our code and models are available at https://github.com/IDEA-Research/DWPose.
Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction
View Paper: https://www.semanticscholar.org/paper/824a19f542ccba4d0b8e83c8a8a6b70f4b7d377b
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
Citations in the corpus
- Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation (2023, methodology): “We employ DWPose[55] to extract pose sequence of characters in the video, including body and hands, rendering it as pose skeleton images following OpenPose[6].”
- Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks (2024, background): “…object detection and segmentation [41, 37, 86], object tracking [67, 84], image generation [75, 54, 48, 45, 23, 14, 52, 22, 82, 44], image editing [42, 1, 2, 53, 21], human-centric perception and understanding [72, 71, 73, 70, 69, 4, 33, 74], and human-centric motion generation [39, 6, 32, 61, 5].”
- VACE: All-in-One Video Creation and Editing (2025, positive): “In the actual training process, the construction for different tasks also needs to be tailored according to the characteristics of each task: 1) For some controllable video generation tasks, we pre-extract depth [51], scribble [6], pose [5, 72], and optical flow [65] from the filtered videos.”
- Champ: Controllable and Consistent Human Image Animation with 3D Parametric Guidance (2024, background): “DWpose [48] stands out as an enhanced alternative to OpenPose [5], offering more accurate and expressive skeletons.”
- MagicAnimate: Temporally Consistent Human Image Animation using Diffusion Model (2023, neutral): “The goal of human image animation is to generate results under the guidance of a reference image I ref .”
- HunyuanVideo: A Systematic Framework For Large Video Generative Models (2024, positive): “We use Dwpose [92] to detect skeletal video from any source video, and use 3DVAE to transform it to latent space as z pose .”