PersonLab: Person Pose Estimation and Instance Segmentation with a Bottom-Up, Part-Based, Geometric Embedding Model

By George Papandreou, Tyler Lixuan Zhu, Liang-Chieh Chen, Spyros Gidaris, Jonathan Tompson, Kevin P. Murphy. 2018

601 citations recorded by the source corpus.

We present a box-free bottom-up approach for the tasks of pose estimation and instance segmentation of people in multi-person images using an efficient single-shot model. The proposed PersonLab model tackles both semantic-level reasoning and object-part associations using part-based modeling. Our model employs a convolutional network which learns to detect individual keypoints and predict their relative displacements, allowing us to group keypoints into person pose instances. Further, we propose a part-induced geometric embedding descriptor which allows us to associate semantic person pixels with their corresponding person instance, delivering instance-level person segmentations. Our system is based on a fully-convolutional architecture and allows for efficient inference, with runtime essentially independent of the number of people present in the scene. Trained on COCO data alone, our system achieves COCO test-dev keypoint average precision of 0.665 using single-scale inference and 0.687 using multi-scale inference, significantly outperforming all previous bottom-up pose estimation systems. We are also the first bottom-up method to report competitive results for the person class in the COCO instance segmentation task, achieving a person category average precision of 0.417.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Alan L. Yuille
2. Liang-Chieh Chen
3. Luc Van Gool
4. Quoc V. Le
5. Dacheng Tao
6. Jingdong Wang
7. Bin Xiao
8. Yi Yang

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Xiaogang Wang
4. Ming-Hsuan Yang
5. Jitendra Malik
6. Alan L. Yuille
7. Yi Yang
8. Trevor Darrell

Citations in the corpus

• Objects as Points (2019, methodology): “To refine the keypoints, we further estimate k human joint heatmaps Φ̂ ∈ RWR ×HR×k using standard bottom-up multi-human pose estimation [4,39,41].”
• Searching for Efficient Multi-Scale Architectures for Dense Image Prediction (2018, background): “A critical element of building such systems is supplying global features and context information to perform pixel-level classification [35, 78, 41, 45, 31, 92, 60, 19, 63].”
• Deep High-Resolution Representation Learning for Visual Recognition (2019, neutral): “t size #Params GFLOPs AP AP50 AP75 APM APL AR Bottom-up: keypoint detection and grouping OpenPose [11] 61:8 84:9 67:5 57:1 68:2 66:5 Associative Embedding [82] 65:5 86:8 72:3 60:6 72:6 70:2 PersonLab [86] 68:7 89:0 75:4 64:1 75:5 75:4 MultiPoseNet [55] 69:6 86:3 76:6 65:0 76:3 73:5 Top-down: human d”
• Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, methodology): “Panoptic-DeepLab is trained with three loss functions: weighted bootstrapped cross entropy loss for semantic segmentation head (Lsem) [75]; MSE loss for center heatmap head (Lheatmap) [67]; and L1 loss for center offset head (Loffset) [58].”
• Deep High-Resolution Representation Learning for Human Pose Estimation (2019, background): “This paper is interested in single-person pose estimation, which is the basis of other related problems, such as multiperson pose estimation [6, 26, 32, 38, 46, 55, 40, 45, 17, 69],”
• Bottom-Up Object Detection by Grouping Extreme and Center Points (2019, methodology): “The network is trained in a fully supervised way, with either an L2 loss to a rendered Gaussian map [3,5,30,49] or with a per-pixel logistic regression loss [22, 34, 35].”
• DeeperLab: Single-Shot Image Parser (2019, methodology): “Similar to [85, 86, 87], we adopt a keypoint-based representation for object instances.”
• PifPaf: Composite Fields for Human Pose Estimation (2019, methodology): “We apply non-maximum suppression at the keypoint level as in [34].”
• Pose Recognition with Cascade Transformers (2021, methodology): “9 − PersonLab [24] ResNet-101 − − − 65.”
• Human Pose Regression with Residual Log-likelihood Estimation (2021, background): “In the bottom-up framework [48, 18, 20, 4, 38, 44, 8], multiple body joints are retrieved from the heatmaps and grouped into different human poses.”
• Monocular human pose estimation: A survey of deep learning-based methods (2020, background): “Papandreou et al. (2018) proposed a box-free multi-task network for pose estimation and instance segmentation.”
• Segmentation-Driven 6D Object Pose Estimation (2018, background): “Note that human pose estimation [10, 44, 33] is also related to global 6D object pose prediction techniques.”
• Whole-Body Human Pose Estimation in the Wild (2020, background): “Bottom-up approaches [8,19,20,21,22,23,37,39,41,45] first detect all the keypoints of every person in images and then group them into individuals.”
• HigherHRNet: Scale-Aware Representation Learning for Bottom-Up Human Pose Estimation (2019, methodology): “1 PersonLab+ [33] 68.”
• CrowdPose: Efficient Crowded Scenes Pose Estimation and a New Benchmark (2018, background): “[38] detect individual joints and predict relative displacements for association.”
• TokenPose: Learning Keypoint Tokens for Human Pose Estimation (2021, background): “Nowadays, heatmap-based fully convolutional neural networks [37, 4, 22, 40, 7, 23, 38, 29] have been dominant solutions due to their high performance.”
• PVNet: Pixel-Wise Voting Network for 6DoF Pose Estimation (2018, methodology): “Our method can handle multiple instances based on the strategy proposed in [40, 28].”