Benchmarking and Error Diagnosis in Multi-instance Pose Estimation

By Matteo Ruggero Ronchi, Pietro Perona. 2017

192 citations recorded by the source corpus.

We propose a new method to analyze the impact of errors in algorithms for multi-instance pose estimation and a principled benchmark that can be used to compare them. We define and characterize three classes of errors - localization, scoring, and background - study how they are influenced by instance attributes and their impact on an algorithm’s performance. Our technique is applied to compare the two leading methods for human pose estimation on the COCO Dataset, measure the sensitivity of pose estimation with respect to instance size, type and number of visible keypoints, clutter due to multiple instances, and the relative score of instances. The performance of algorithms, and the types of error they make, are highly dependent on all these variables, but mostly on the number of keypoints and the clutter. The analysis and software tools we propose offer a novel and insightful approach for understanding the behavior of pose estimation algorithms and an effective method for measuring their strengths and weaknesses.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Whole-Body Human Pose Estimation in the Wild (2020, methodology): “, we also develop a tool for deeper performance analysis based on [48] which will be provided to facilitate offline evaluation.”
Through-Wall Human Pose Estimation Using Radio Signals (2018, background): “It has recently witnessed major advances and significant performance improvements [30, 27, 28, 46, 31, 20, 10, 16, 33, 12, 47, 37, 45, 13].”
Diagnosing Error in Temporal Action Detectors (2018, methodology): “For example, we are still not able to answer the following questions: How close are we to achieve our goal of delimiting the start and end of actions? What makes an algorithm more effective than another? What makes an action hard to localize? Is the uncertainty of the temporal boundaries impeding th”
Pose2Mesh: Graph Convolutional Network for 3D Human Pose and Mesh Recovery from a 2D Human Pose (2020, background): “However, the estimated 2D pose often contains errors [50], especially under severe occlusions or challenging poses.”
StarMap for Category-Agnostic Keypoint and Viewpoint Estimation (2018, background): “We can see our score is 92 .”
AI Choreographer: Music Conditioned 3D Dance Generation with AIST++ (2021, background): “• 17 COCO-format[70] human joint locations in both 2D and 3D;”
SLEAP: A deep learning system for multi-animal pose tracking (2022, methodology): “To estimate the overall multi-animal accuracy, we calculated the object keypoint similarity (OKS) score, an accuracy metric that takes into account the uncertainty in landmark localization, visibility and animal scale 17 .”