Towards Accurate Multi-person Pose Estimation in the Wild

By George Papandreou, Tyler Lixuan Zhu, Nori Kanazawa, Alexander Toshev, Jonathan Tompson, C. Bregler, Kevin P. Murphy. 2017

872 citations recorded by the source corpus.

We propose a method for multi-person detection and 2-D pose estimation that achieves state-of-art results on the challenging COCO keypoints task. It is a simple, yet powerful, top-down approach consisting of two stages. In the first stage, we predict the location and scale of boxes which are likely to contain people, for this we use the Faster RCNN detector. In the second stage, we estimate the keypoints of the person potentially contained in each proposed bounding box. For each keypoint type we predict dense heatmaps and offsets using a fully convolutional ResNet. To combine these outputs we introduce a novel aggregation procedure to obtain highly localized keypoint predictions. We also use a novel form of keypoint-based Non-Maximum-Suppression (NMS), instead of the cruder box-level NMS, and a novel form of keypoint-based confidence score estimation, instead of box-level scoring. Trained on COCO data alone, our final system achieves average precision of 0.649 on the COCO test-dev set and the 0.643 test-standard sets, outperforming the winner of the 2016 COCO keypoints challenge and other recent state-of-art. Further, by using additional in-house labeled data we obtain an even higher average precision of 0.685 on the test-dev set and 0.673 on the test-standard set, more than 5% absolute improvement compared to the previous best performing method on the same dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Exploring the Limits of Weakly Supervised Pretraining (2018, background): “Examples include: object detection [1,4], semantic segmentation [5,6], human pose estimation [7,8], video recognition [9], monocular depth estimation [10], and so on.”
Revisiting Unreasonable Effectiveness of Data in Deep Learning Era (2017, methodology): “These visual representations have been shown to be effective on a wide range of computer vision tasks [1, 4, 14, 22, 29, 33, 36].”
Objects as Points (2019, methodology): “We use the default output stride of R = 4 in literature [4,40,42].”
Deep High-Resolution Representation Learning for Visual Recognition (2019, methodology): “The two-stage top-down paradigm similar as [24], [109], [152] is used: detect the person instance using a person detector, and then predict detection keypoints.”
Deep High-Resolution Representation Learning for Human Pose Estimation (2019, methodology): “The two-stage top-down paradigm similar as [46, 11, 70] is used: detect the person instance using a person detector, and then predict detection keypoints.”
Survey on Emotional Body Gesture Recognition (2018, background): “tion has shifted towards multi-person pose estimation [74], [78], [79].”
Benchmarking and Error Diagnosis in Multi-instance Pose Estimation (2017, methodology): “Although the AP improvement after rescoring is equivalent, [29] provides scores that are in the same order as the optimal ones for a higher percentage of images and makes less errors, indicating that it is using a better scoring function.”
A Survey of Deep Learning-Based Object Detection (2019, methodology): “The best performing methods [292][9][293][294] typically base on one-stage backbone networks.”
OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields (2018, methodology): “Multi-Person Pose Estimation For multi-person pose estimation, most approaches [5], [6], [38], [39], [40], [41], [42], [43], [44] have used a top-down strategy that first detects people and then have estimated the pose of each person independently on each detected region.”
LCR-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images (2018, methodology): “[22] also compute a per-joint regressor at each pixel to refine the position of the joints, that may lack precision due to the stride of CNNs.”
Pose Recognition with Cascade Transformers (2021, methodology): “Heatmap-based approaches include [2, 13, 25, 21, 17, 24, 6, 5, 37, 28, 40, 39, 30] where various techniques have been developed to perform multi-class keypoint classification.”
Camera Distance-Aware Top-Down Approach for 3D Multi-Person Pose Estimation From a Single RGB Image (2019, background): “[6,13,30,31,34,47] are based on the top-down approach.”
Deep Learning-based Human Pose Estimation: A Survey (2020, methodology): “[79] proposed a two-stage architecture, consisting of a Faster R-CNN person detector to create bounding boxes for candidate human bodies and a keypoint estimator to predict the locations of keypoints by using a form of heatmap-offset aggregation.”
From Synthetic to Real: Unsupervised Domain Adaptation for Animal Pose Estimation (2021, background): “Recently, deep learning based approaches [28, 24, 8, 39, 35, 7, 38, 26] have achieved significant progress due to the availability of large scale training data such as the MPII dataset [1] and the COCO keypoint detection dataset [21].”