Multi-context Attention for Human Pose Estimation

By Xiao Chu, Wei Yang, Wanli Ouyang, Cheng Ma, Alan L. Yuille, Xiaogang Wang. 2017

689 citations recorded by the source corpus.

In this paper, we propose to incorporate convolutional neural networks with a multi-context attention mechanism into an end-to-end framework for human pose estimation. We adopt stacked hourglass networks to generate attention maps from features at multiple resolutions with various semantics. The Conditional Random Field (CRF) is utilized to model the correlations among neighboring regions in the attention map. We further combine the holistic attention model, which focuses on the global consistency of the full human body, and the body part attention model, which focuses on detailed descriptions for different body parts. Hence our model has the ability to focus on different granularity from local salient regions to global semantic consistent spaces. Additionally, we design novel Hourglass Residual Units (HRUs) to increase the receptive field of the network. These units are extensions of residual units with a side branch incorporating filters with larger receptive field, hence features with various scales are learned and combined within the HRUs. The effectiveness of the proposed multi-context attention mechanism and the hourglass residual units is evaluated on two widely used human pose estimation benchmarks. Our approach outperforms all existing methods on both benchmarks over all the body parts. Code has been made publicly available.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, methodology): “[16] proposed an attention model based on conditional random field (CRF) and Yang et al.”
Deep High-Resolution Representation Learning for Visual Recognition (2019, methodology): “…process (e.g., VGGNet), with skipping connection over some mirrored layers to transform the pooling indices, e.g., SegNet [3] and DeconvNet [107], or copying the feature maps, e.g., U-Net [119] and Hourglass [8], [9], [27], [31], [68], [105], [134], [163], [165], encoder-decoder [112], and so on.”
Compositional Human Pose Regression (2017, methodology): “[11] further extend [31] with a multi-context attention mechanism.”
Deep High-Resolution Representation Learning for Human Pose Estimation (2019, methodology): “Following [14, 74, 60], a six-scale pyramid testing procedure is performed.”
Monocular human pose estimation: A survey of deep learning-based methods (2020, background): “In terms of iterative reﬁnement, some work designed networks in a multi-stage style to reﬁne results from coarse prediction via end-to-end learning (Tompson et al., 2015; Bulat and Tzimiropoulos, 2016; Newell et al., 2016; Wei et al., 2016; Yang et al., 2017; Belagiannis and Zisserman, 2017).”
Dense Recurrent Neural Networks for Scene Labeling (2018, methodology): “[11] propose to utilize attention model to combine multi-context for human pose estimation.”
Spatial Temporal Transformer Network for Skeleton-based Action Recognition (2020, background): “Moreover, recent advances in 3D depth cameras such as Microsoft Kinect sensors [1], [2] and advanced human pose estimation algorithms [3], [4] made it possible to estimate 3D skeleton coordinates quickly and accurately with cheap devices.”
Simple Baselines for Human Pose Estimation and Tracking (2018, methodology): “Hourglass [4] is the dominant approach on MPII benchmark as it is the basis for all leading methods [5,6,7,8].”
Multi-Scale Self-Guided Attention for Medical Image Segmentation (2019, background): “Alternatively, attention mechanisms have been widely studied in deep CNNs for many computer vision tasks in order to efficiently integrate local and global features, including human pose estimation [12], emotion recognition [13], text detection [14], object detection [15] and classification [16].”
OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields (2018, methodology): “To obtain reliable local observations of body parts, Convolutional Neural Networks (CNNs) have been widely used, and have significantly boosted the accuracy on body pose estimation [19], [20], [21], [22], [23], [24], [25], [26], [27], [28], [29], [30], [31], [32].”
Adversarial PoseNet: A Structure-Aware Convolutional Network for Human Pose Estimation (2017, neutral): “9 [7] 98.”
Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison (2019, methodology): “There are two mainstream approaches: regressing the key-point positions [65, 11], and estimating keypoint heatmaps followed by a non-maximal suppression technique [9, 19, 18, 73].”
Integral Human Pose Regression (2017, methodology): “The most widely adopted is mean square error (or L2 distance) between the predicted heat map and ground-truth heat map with a 2D Gaussian blob centered on the ground truth joint location [41, 43, 5, 29, 8, 10, 12, 4].”
Deep Learning-based Human Pose Estimation: A Survey (2020, background): “Max Planck Institute for Informatics (MPII) Year Method Head Shoulder Elbow Wrist Hip Knee Ankle Mean Params(M) FLOPs(G) 2017 [42] 98.”
Attentional Feature Fusion (2020, background): “In the ﬁrst type, the features at multiple scales or their concatenated result are fed into the attention module to generate multi-scale attention maps, while the scale of feature context aggregation inside the attention module remains single [2, 3, 46, 6, 36, 41].”
Human Pose Regression by Combining Indirect Part Detection and Contextual Information (2017, neutral): “Similar techniques have been used by many previous works [5, 27, 13], resulting in signiﬁcant gain of performance.”
ADCrowdNet: An Attention-Injective Deformable Convolutional Network for Crowd Understanding (2018, background): “This principle of focusing on specific parts of the input has been successfully applied in various deep learning models for images classification [11], semantic segmentation [24], image deblurring [22], and visual pose estimation [6], which also suits our problem where the interest regions containin”
Fast Human Pose Estimation (2018, background): “Human Pose Estimation The past ﬁve years have witnessed a huge progress of human pose estimation in the deep learning regime [30, 28, 6, 31, 19, 11, 32, 20, 22].”
Look into Person: Joint Body Parsing & Pose Estimation Network and a New Benchmark (2018, background): “Some studies [29], [30] explored encoding part constraints and contextual information for guiding the network to focus on informative regions (human parts) to predict more precise locations”