OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields

By Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, Yaser Sheikh. 2018

5,711 citations recorded by the source corpus.

Realtime multi-person 2D pose estimation is a key component in enabling machines to have an understanding of people in images and videos. In this work, we present a realtime approach to detect the 2D pose of multiple people in an image. The proposed method uses a nonparametric representation, which we refer to as Part Affinity Fields (PAFs), to learn to associate body parts with individuals in the image. This bottom-up system achieves high accuracy and realtime performance, regardless of the number of people in the image. In previous work, PAFs and body part location estimation were refined simultaneously across training stages. We demonstrate that a PAF-only refinement rather than both PAF and body part location refinement results in a substantial increase in both runtime performance and accuracy. We also present the first combined body and foot keypoint detector, based on an internal annotated foot dataset that we have publicly released. We show that the combined detector not only reduces the inference time compared to running them sequentially, but also maintains the accuracy of each component individually. This work has culminated in the release of OpenPose, the first open-source realtime system for multi-person 2D pose detection, including body, foot, hand, and facial keypoints.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Objects as Points (2019, background): “Object detection is then a standard keypoint estimation problem [3,39,60].”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, methodology): “(d) Consequently, we detect hand joints with OpenPose [8] and optimize hand shape using Eq.”
Reconstructing Hand-Object Interactions in the Wild (2020, methodology): “We further optimize the hand pose by a fitting procedure which uses 2D hand keypoints from OpenPose [6, 51] The hand pose optimization objective is to minimize the difference between 2D keypoints detection and 2D projection of 3D hand keypoints [24, 1]:”
Core Challenges of Social Robot Navigation: A Survey (2021, background): “While existing approaches [27] may provide such functionalities, their performance is highly dependent on the sensing hardware of the robot and the conditions in the robot’s environment.”
Deep 3D human pose estimation: A review (2021, neutral): “Skeleton-Based Model: First and foremost, the skeleton model is commonly used in 2D human pose estimation (Cao et al., 2018) and is naturally extended to 3D.”
AI Choreographer: Music Conditioned 3D Dance Generation with AIST++ (2021, background): “Training data for 2D pose and audio is abundant thanks to the high reliability of 2D pose detectors [14].”
Towards Multimodal Sarcasm Detection (An _Obviously_ Perfect Paper) (2019, background): “Advanced techniques to learn multimodal relationships could incorporate better relationship modeling (Majumder et al., 2018), and exploit models that provide gesture, facial and pose information about the people in the scene (Cao et al., 2018).”
Dynamic GCN: Context-enriched Topology Learning for Skeleton-based Action Recognition (2020, methodology): “Kinetics) or estimated with human pose estimation algorithms [4, 5, 31].”
AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time (2022, methodology): “Our saving format is COCO format by default and can be compatible with OpenPose [17].”
Soft Rasterizer: A Differentiable Renderer for Image-Based 3D Reasoning (2019, methodology): “By detecting the 2D key points, great progress has been made to estimate the 2D poses [31, 4, 45].”
VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization (2021, methodology): “We ﬁrst predict the segmentation map S ∈ L H × W and the pose map P ∈ 3 × H × W of the image I by utilizing the pre-trained networks [7, 3], where L is a set of integers indicating the semantic labels.”
High-Order Information Matters: Learning Relation and Topology for Occluded Person Re-Identification (2020, methodology): “Following the ideas above and inspired by recent developments on person ReID [43, 35, 24, 4] and human key-points prediction [2, 33], we utilize a CNN backbone to extract local features of different key-points.”
A model-based gait recognition method with body pose and human prior knowledge (2020, methodology): “Recently, a ottom-up pose estimation method [6] using deep CNN can create ccurate human models.”
Whole-Body Human Pose Estimation in the Wild (2020, methodology): “By contrast, the model complexity of OpenPose [7] is 451.”
Word-level Deep Sign Language Recognition from Video: A New Large-scale Dataset and Methods Comparison (2019, methodology): “There are two mainstream approaches: regressing the keypoint positions [65, 11], and estimating keypoint heatmaps followed by a non-maximal suppression technique [9, 19, 18, 73].”
VIBE: Video Inference for Human Body Pose and Shape Estimation (2019, methodology): “PennAction [69] and PoseTrack [3] are the only ground-truth 2D video datasets we use, while InstaVariety [30] and Kinetics-400 [31] are pseudo ground-truth datasets annotated using a 2D keypoint detector [12, 35].”
Monocular Total Capture: Posing Face, Body, and Hands in the Wild (2018, methodology): “The bounding box is given by OpenPose2 [15, 51, 14] for testing.”
Effective Whole-body Pose Estimation with Two-stages Distillation (2023, background): “We also compare our state-of-the-art model with two widely used models OpenPose [2] and MediaPipe [29, 62], as presented in Fig.”