HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation

By Bowen Cheng, Bin Xiao, Jingdong Wang, Humphrey Shi, Thomas S. Huang, Lei Zhang. Computer Vision and Pattern Recognition, 2019

832 citations recorded by the source corpus.

Bottom-up human pose estimation methods have difficulties in predicting the correct pose for small persons due to challenges in scale variation. In this paper, we present HigherHRNet: a novel bottom-up human pose estimation method for learning scale-aware representations using high-resolution feature pyramids. Equipped with multi-resolution supervision for training and multi-resolution aggregation for inference, the proposed approach is able to solve the scale variation challenge in bottom-up multi-person pose estimation and localize keypoints more precisely, especially for small person. The feature pyramid in HigherHRNet consists of feature map outputs from HRNet and upsampled higher-resolution outputs through a transposed convolution. HigherHRNet outperforms the previous best bottom-up method by 2.5% AP for medium person on COCO test-dev, showing its effectiveness in handling scale variation. Furthermore, HigherHRNet achieves new state-of-the-art result on COCO test-dev (70.5% AP) without using refinement or other post-processing techniques, surpassing all existing bottom-up methods. HigherHRNet even surpasses all top-down methods on CrowdPose test (67.6% AP), suggesting its robustness in crowded scene.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f6dfd6a73442018b223aa904d73e82a8f68a374b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression (2021, methodology): “The bottom-up paradigm either directly re-gresses the keypoint positions belonging to the same person, or detects and groups the keypoints, such as afﬁnity linking [7, 31], associative embedding [40], HGG [27] and HigherHRNet [11].”
Semi-supervised Medical Image Segmentation through Dual-task Consistency (2020, methodology): “2020) and key-points estimation (Cheng et al. 2020) as long as there exists differentiable transform between two tasks.”
YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss (2022, methodology): “Chen et al. proposed Higher HRNet [9] which uses higher output resolution to improve the precision of the prediction by a large margin.”
Pose Recognition with Cascade Transformers (2021, methodology): “An alternative strategy is called a bottom-up process where human keypoints are detected directly from the image without an explicit object detection stage [6].”
Revisiting Skeleton-based Action Recognition (2021, methodology): “Compared to its 2D Bottom-Up counterparts [5, 8, 43], Top-Down methods obtain superior performance on standard benchmarks such as COCO-keypoints [35].”
Multi-animal pose estimation, identification and tracking with DeepLabCut (2022, background): “To associate detected keypoints to particular individuals (assembly) several solutions have been proposed, such as part affinity fields (PAFs) 9 , associative embeddings 10,11 , transformers 12 and other mechanisms 13,14 .”
DS-TransUNet: Dual Swin Transformer U-Net for Medical Image Segmentation (2021, background): “Multi-scale feature representations based on CNNs are a classic concept in computer vision, and have shown to beneﬁt various CV tasks [28] [29] [30] [31].”
Human Pose Regression with Residual Log-likelihood Estimation (2021, background): “In the bottom-up framework [48, 18, 20, 4, 38, 44, 8], multiple body joints are retrieved from the heatmaps and grouped into different human poses.”
Lite-HRNet: A Lightweight High-Resolution Network (2021, background): “EfficientHRNet [30] introduces the mobile convolutions into HigherHRNet [8].”
TransPose: Keypoint Localization via Transformer (2020, methodology): “Further, sophisticated strategies are proposed such as multi-scale fusion [38, 43, 63, 12, 51, 15, 13], stacking [60, 61, 38], or high-resolution representation [51]; meanwhile, many successful architectures have emerged such as CPM [60], Hourglass Network [38], FPN [63], CPN [12], SimpleBaseline [6”
AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time (2022, background): “Cheng et al.[27] use a powerful multi-resolution network[28] as backbone and high-resolution feature pyramids to learn scale-aware representations.”
RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose (2023, background): “Bottom-up algorithms [7, 10, 19, 27, 29, 41, 44, 45] detect instance-agnostic keypoints in an image and partition these keypoints to obtain the human pose.”
Monocular, One-stage, Regression of Multiple 3D People (2020, background): “6M [14] and MPI-INF-3DHP [34]), one pseudo-label 3D dataset (UP [26]) and four in-the-wild 2D pose datasets (MS COCO [30], MPII [2], LSP [16, 17] and AICH [50]).”
FairMOT: On the Fairness of Detection and Re-identification in Multiple Object Tracking (2020, methodology): “Besides DLA, other deep networks that provide multi-scale convolutional features, such as Higher HRNet (Cheng et al. 2020), can be used in our framework to provide fair features for both detection and re-ID.”
Gaitgraph: Graph Convolutional Network for Skeleton-Based Gait Recognition (2021, methodology): “Compared to silhouette images, multiple poses can be extracted from an image simultaneously, even if they overlap [8].”
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification (2021, background): “have proven beneficial for many vision tasks [5, 4, 22, 21, 25, 24, 7], but such potential benefit for vision transformers remains to be validated.”
NeuMan: Neural Human Radiance Field from a Single Video (2022, methodology): “Therefore, we refine the SMPL estimates by optimizing the SMPL parameters using silhouette estimated from [8,48], and 2D joints estimated from [3,4] as detailed in the supplementary material.”