Multi-Scale Structure-Aware Network for Human Pose Estimation

By Lipeng Ke, Ming-Ching Chang, Honggang Qi, Siwei Lyu. European Conference on Computer Vision, 2018

298 citations recorded by the source corpus.

We develop a robust multi-scale structure-aware neural network for human pose estimation. This method improves the recent deep conv-deconv hourglass models with four key improvements: (1) multi-scale supervision to strengthen contextual feature learning in matching body keypoints by combining feature heatmaps across scales, (2) multi-scale regression network at the end to globally optimize the structural matching of the multi-scale features, (3) structure-aware loss used in the intermediate supervision and at the regression to improve the matching of keypoints and respective neighbors to infer a higher-order matching configurations, and (4) a keypoint masking training scheme that can effectively fine-tune our network to robustly localize occluded keypoints via adjacent matches. Our method can effectively improve state-of-the-art pose estimation methods that suffer from difficulties in scale varieties, occlusions, and complex multi-person scenarios. This multi-scale supervision tightly integrates with the regression network to effectively (i) localize keypoints using the ensemble of multi-scale features, and (ii) infer global pose configuration by maximizing structural consistencies across multiple keypoints and scales. The keypoint masking training enhances these advantages to focus learning on hard occlusion samples. Our method achieves the leading position in the MPII challenge leaderboard among the state-of-the-art methods.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4ca04fe5e71afe6f6b217ac7bb04c678604c171c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “[97] designed a multi-scale structure-aware neural network, which combines multi-scale supervision, multi-scale feature combination, structure-aware loss information scheme, and a keypoint masking training method to improve HPE models in complex scenarios.”
Toward fast and accurate human pose estimation via soft-gated skip connections (2020, methodology): “Current state-of-the-art on single person pose estimation is held by variants of the so-called HourGlass [23], [38], [3], [19], [8], [39], [6] and U-Net architectures [28], [33], [35].”
Deep High-Resolution Representation Learning for Human Pose Estimation (2019, methodology): “Hourglass [39] and its extensions [74, 30] combine low-level features in the high-to-low process into the same-resolution high-level features in the low-to-high process progressively through skip connections.”
SPGNet: Semantic Prediction Guidance for Scene Parsing (2019, background): “On the other hand, the multi-stage encoder-decoder models [5, 28, 33, 41, 42, 61, 68], also known as stacked hourglass networks [42], reﬁne the keypoint estimation iteratively by propagating information across stages for the task of human pose estimation.”
Distribution-Aware Coordinate Representation for Human Pose Estimation (2019, background): “…active learning (Liu and Ferrari 2017), adversarial learning (Chen et al. 2017), deconvolution upsampling (Xiao, Wu, and Wei 2018), multi-scale supervision (Ke et al. 2018), attentional mechanism (Liu et al. 2018; Su et al. 2019), and high-resolution representation preserving (Sun et al. 2019).”
OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields (2018, methodology): “To obtain reliable local observations of body parts, Convolutional Neural Networks (CNNs) have been widely used, and have signiﬁcantly boosted the accuracy on body pose estimation [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32].”
Deep High-Resolution Representation Learning for Visual Recognition (2019, methodology): “…process (e.g., VGGNet), with skipping connection over some mirrored layers to transform the pooling indices, e.g., SegNet [3] and DeconvNet [107], or copying the feature maps, e.g., U-Net [119] and Hourglass [8], [9], [27], [31], [68], [105], [134], [163], [165], encoder-decoder [112], and so on.”