Hand Keypoint Detection in Single Images Using Multiview Bootstrapping

By Tomas Simon, Hanbyul Joo, Iain Matthews, Yaser Sheikh. 2017

1,225 citations recorded by the source corpus.

We present an approach that uses a multi-camera system to train fine-grained detectors for keypoints that are prone to occlusion, such as the joints of a hand. We call this procedure multiview bootstrapping: first, an initial keypoint detector is used to produce noisy labels in multiple views of the hand. The noisy detections are then triangulated in 3D using multiview geometry or marked as outliers. Finally, the reprojected triangulations are used as new labeled training data to improve the detector. We repeat this process, generating more labeled data in each iteration. We derive a result analytically relating the minimum number of views to achieve target true and false positive rates for a given detector. The method is used to train a hand keypoint detector for single images. The resulting keypoint detector runs in realtime on RGB images and has accuracy comparable to methods that use depth sensors. The single view detector, triangulated over multiple views, enables 3D markerless hand motion capture with complex object interactions.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Simitate: A Hybrid Imitation Learning Benchmark (2019, neutral): “works that recently opened up possibilities for reasonable accurate object recognition [1], [2], [3], detection [4], [5], semantic segmentation [6], [7], [8] and human pose estimations [9], [10], [11], we believe in advancing these fundamental approaches to actual scene understanding and”
Data Distillation: Towards Omni-Supervised Learning (2017, methodology): “[38] utilize multi-view geometry to generate hand keypoint labels from multiple cameras and retrain the detector.”
Expressive Body Capture: 3D Hands, Face, and Body From a Single Image (2019, methodology): “OpenPose [15, 60, 70] expands this to include the 2D hand joints and 2D facial features.”
Weakly-Supervised Mesh-Convolutional Hand Reconstruction in the Wild (2020, methodology): “Hand Pose Estimation Multiview bootstrapping [45] is an iterative improvement technique where the initial annotations of a hand from multiple views, obtained by executing a 2D keypoint detector, are triangulated in 3D and reprojected for training in the next iteration if they meet the quality criter”
Energy-Based Models for Deep Probabilistic Regression (2020, methodology): “The other approach is to treat the confidence prediction as a direct regression problem itself by applying standard regression losses, such as L(2) [50, 8, 55] or the Huber loss [23].”
DexYCB: A Benchmark for Capturing Hand Grasping of Objects (2021, background): “To acquire 3D pose at scale others rely on automatic or semi-automatic approaches [30, 50].”
3D Hand Shape and Pose From Images in the Wild (2019, methodology): “MPII+NZSL [44]) compared to datasets with 3D ground-truth, thus helping improve generalization to this type of challenging data.”
Leveraging Photometric Consistency Over Time for Sparsely Supervised Hand-Object Reconstruction (2020, neutral): “Multi-view setups [43, 56], which enable 3D triangulation from 2D detections, can similarly only be captured in controlled environments.”
ARCTIC: A Dataset for Dexterous Bimanual Hand-Object Manipulation (2022, background): “[57] Tomas Simon, Hanbyul Joo, Iain Matthews, and Yaser Sheikh.”
Grasping Field: Learning Implicit Representations for Human Grasps (2020, background): “, RGB images [2, 15, 32, 54, 60, 79, 80, 96] or RGB-D and depth sensors [24, 35, 48, 55, 57].”
Learnable Triangulation of Human Pose (2019, methodology): “The CMU panoptic is a new multi-camera dataset maintained by the Carnegie Mellon University [5, 20, 15]. cameras acquired via triangulation using all camera views.”
3D Hand Shape and Pose Estimation From a Single RGB Image (2019, methodology): “However, existing hand RGB image datasets [43, 62, 63, 33] only provide the annotations of 2D/3D hand joint locations, and they do not contain any 3D hand shape annotations.”
Single-Shot Multi-person 3D Pose Estimation from Monocular RGB (2017, methodology): “Some prior work creates 3D annotated multi-person images by using 2D pose data augmented with 3D poses from motion capture datasets [49], or by ﬁnding 3D consistency in 2D part annotations from multi-view images recorded in a studio [53].”
A Hand Motion-guided Articulation and Segmentation Estimation (2020, methodology): “To detect the hand position in RGB images, we use hand keypoint detection [18] implemented in OpenPose [15].”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, background): “While a significant amount of research has focused on predicting the pose of hands [27, 54, 56, 58, 70, 93, 94, 98] or objects [5, 48, 59, 78, 83, 90] in isolation, joint understanding of handobject interactions has received far less attention.”
ContactPose: A Dataset of Grasps with Object Contact and Hand Pose (2020, methodology): “For each RGB frame, we use Detectron [26] to locate the wrist, and run the OpenPose hand keypoint detector [53] on a 200×200 crop around the wrist.”
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person Perspectives (2023, neutral): “Some methods assume no body visibility [63, 80, 98, 187, 188], while others assume partial observability by modifying cameras to capture the body [3, 57, 136, 161, 181].”
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration (2020, methodology): “We compare our hand module with the previous state-of-the-art hand approaches on three public hand benchmarks, STB [59], RHD [62] and MPII+NZSL [46].”