Spatial Attention Deep Net with Partial PSO for Hierarchical Hybrid Hand Pose Estimation

By Qianru Ye, Shanxin Yuan, Tae-Kyun Kim. 2016

160 citations recorded by the source corpus.

Discriminative methods often generate hand poses kinematically implausible, then generative methods are used to correct (or verify) these results in a hybrid method. Estimating 3D hand pose in a hierarchy, where the high-dimensional output space is decomposed into smaller ones, has been shown effective. Existing hierarchical methods mainly focus on the decomposition of the output space while the input space remains almost the same along the hierarchy. In this paper, a hybrid hand pose estimation method is proposed by applying the kinematic hierarchy strategy to the input space (as well as the output space) of the discriminative method by a spatial attention mechanism and to the optimization of the generative method by hierarchical Particle Swarm Optimization (PSO). The spatial attention mechanism integrates cascaded and hierarchical regression into a CNN framework by transforming both the input (and feature space) and the output space, which greatly reduces the viewpoint and articulation variations. Between the levels in the hierarchy, the hierarchical PSO forces the kinematic constraints to the results of the CNNs. The experimental results show that our method significantly outperforms four state-of-the-art methods and three baselines on three public benchmarks.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, 3D Vision & Pose, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In 3D Vision & Pose

In Human-Computer Interaction

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
Hand Keypoint Detection in Single Images Using Multiview Bootstrapping (2017, background): “These are broadly classifiable into generative methods [14], discriminative methods [24, 25, 9, 30, 22, 27], or hybrid methods [20, 16, 18, 26, 31].”
Reconstructing Hand-Object Interactions in the Wild (2020, background): “Many previous works on hand pose estimation directly predict 3D joint locations from either depth [50, 53, 55, 59, 64, 65, 35] or RGB [45, 36, 4, 63, 68] images.”
V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map (2017, methodology): “[52] used spatial attention mechanism and hierarchical PSO.”
Real-Time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor (2017, background): “Recent deep learning methods promise large learning capacities for hand pose estimation [39, 4, 29, 48, 46, 18].”
Depth-Based Hand Pose Estimation: Methods, Data, and Challenges (2015, unknown): “For highly precise hand pose estimation, recent hybrid pipelines compliment data-driven per-frame reinitialization with model-based refinement (Taylor et al. 2016; Ballan et al. 2012; Sridhar et al. 2015; Qian et al. 2014; Ye et al. 2016).”
First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations (2017, background): “3D hand pose estimation: Mainly due to the recent availability of RGB-D sensors, the field has made significant progress in object-less third-person view [22, 25, 28, 37, 39, 41, 45, 53, 60, 80] and more modest advances in first-person view [10, 35, 38, 48].”
Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals (2017, background): “2D CNNs have been popular for 3D hand pose estimation [1, 3, 14, 15, 20, 21, 23, 29, 57, 61].”
Total Capture: A 3D Deformation Model for Tracking Faces, Hands, and Bodies (2018, background): “Hand motion captures are mostly lead by single depth sensor based methods [36, 46, 49, 30, 57, 45, 53, 43, 39, 42, 50, 58], with few exceptions based on multi-view systems [4, 43, 38].”
GANerated Hands for Real-Time 3D Hand Tracking from Monocular RGB (2017, background): “Hybrid methods that combine the best of both generative and discriminative methods show the best performance on benchmark datasets [47, 39, 40, 20, 59].”
HOnnotate: A Method for 3D Annotation of Hand and Object Poses (2019, background): “Generative approaches take advantage of a hand model and its kinematic structure to generate hand pose hypotheses that are physically plausible [12, 28, 45, 51, 53, 63, 70].”
FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration (2020, methodology): “Similarly, there have been noticeable achievements in 3D hand pose estimation from a depth input [34, 48, 45, 47, 53, 58] or a single RGB image [62, 9, 15, 8, 11, 61].”
Monocular Total Capture: Posing Face, Body, and Hands in the Wild (2018, unknown): “Most of previous work is based on depth image as input [40, 54, 49, 52, 65, 71].”
Cross-Modal Deep Variational Hand Pose Estimation (2018, background): “Furthermore, a number of hybrid methods exist that first leverage a discriminative model to initialize a hand pose estimate which is then refined and tracked via carefully designed energy functions to fit a hand model into the observed depth data [19, 22, 30, 33, 36].”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, background): “While a significant amount of research has focused on predicting the pose of hands [27, 54, 56, 58, 70, 93, 94, 98] or objects [5, 48, 59, 78, 83, 90] in isolation, joint understanding of handobject interactions has received far less attention.”