Training a Feedback Loop for Hand Pose Estimation

By Markus Oberweger, Paul Wohlhart, Vincent Lepetit. 2015

283 citations recorded by the source corpus.

We propose an entirely data-driven approach to estimating the 3D pose of a hand given a depth image. We show that we can correct the mistakes made by a Convolutional Neural Network trained to predict an estimate of the 3D pose by using a feedback loop. The components of this feedback loop are also Deep Networks, optimized using training data. They remove the need for fitting a 3D model to the input data, which requires both a carefully designed fitting function and algorithm. We show that our approach outperforms state-of-the-art methods, and is efficient as our implementation runs at over 400 fps on a single GPU.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

End-to-End Recovery of Human Shape and Pose (2017, methodology): “In [7, 31] the estimates are rendered to an image space to concatenate with the image input.”
Human Pose Estimation with Iterative Error Feedback (2015, background): “In concurrent work, Oberweger et al [27] proposed a feedback loop for hand pose estimation from kinect data that is closely related to our approach.”
Recurrent Human Pose Estimation (2016, background): “to the architecture of [26] which also switches between direct pose coordinates and an image representation in an iterated module.”
SurfNet: Generating 3D Shape Surfaces Using Deep Residual Networks (2017, methodology): “This correction mechanism hints that we can use feedback from the neural network to rectify noisy correspondences in the training set and also incorporate additional models for training a neural network, similar in spirit to [21].”
Geometric Back-Projection Network for Point Cloud Classification (2019, background): “ stable and responsive systems in industry. Some early works managed to involve this mechanism in CNNs to deal with computer vision problems like 2D visual segmentation [5] or 3D hand pose estimation [24] etc. However, for point clouds, PU-GAN [13] to some extent used a feedback-like unit but for fe”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, background): “A significant amount of research has focused on visual understanding of humans [8, 25, 36, 40, 59, 65, 70, 74] and objects [4, 28, 64, 66], in isolation from each other.”
3D Hand Shape and Pose From Images in the Wild (2019, background): “There is a rich literature on 3D hand pose and reconstruction from depth [62, 66, 10, 36, 61, 11, 43, 45, 19, 20, 24, 30, 37, 48, 52, 53, 59, 64], image and depth [26, 32, 49, 28], stereo [33, 65, 41] and multiple images [8, 31, 50].”
V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map (2017, methodology): “Most of the existing methods for 3D pose estimation from a single depth map [1, 2, 6, 13–15, 23, 28–30, 46] are based on the model in Figure 2(a) that takes a 2D depth image and directly regresses 3D coordinates.”
First-Person Hand Action Benchmark with RGB-D Videos and 3D Hand Pose Annotations (2017, background): “the-art approaches in Sections4and5. Hand pose estimation: Mainly due to the availability of commodity RGB-D sensors, hand pose estimation has recently made signiﬁcant progresses in third-person view [18,21,23,29,31,33,37,45,51,69] and more modest advances in ﬁrst-person setting [30,40,41]. In [34],”
BB8: A Scalable, Accurate, Robust to Partial Occlusion Method for Predicting the 3D Poses of Challenging Objects without Using Depth (2017, methodology): “The second image depends on the current estimate of the pose: While [17] generates a depth map with a deep network, we render (using OpenGL) either a binary mask or a color rendering of the target object as seen from this current estimate.”
Embodied hands (2017, methodology): “…Ye et al. 2013] in light of the recent advancements in consumer RGB-D sensors and virtual reality [Hamer et al. 2009; Leap 2017; Melax et al. 2013; Oberweger et al. 2015; Oikonomidis et al. 2011a, 2012; Romero et al. 2013; Schmidt et al. 2014; Schröder et al. 2014; Sridhar et al. 2015;…”
Hand Keypoint Detection in Single Images Using Multiview Bootstrapping (2017, methodology): “[13] use feedback loops to generate synthetic training data for hand pose estimation, motivated by the same principles as our work, but focus on generating depth images.”
Robust 3D Hand Pose Estimation in Single Depth Images: From Single-View CNN to Multi-View CNNs (2016, methodology): “Index Terms—3D hand pose estimation, Convolutional neural networks, Multi-view CNNs F”
Learning Joint Reconstruction of Hands and Manipulated Objects (2019, unknown): “The availability of commodity RGB-D sensors [21, 43, 54] led to significant progress in estimating 3D hand pose given depth or RGB-D input [13, 20, 34, 35].”
PyMAF: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop (2021, unknown): “To put our approach in a broader view, there have been remarkable efforts made to involve iterative fitting strategies in other computer vision tasks, including facial landmark localization [57, 53], human/hand pose estimation [40, 7], etc.”
Monocular Real-Time Hand Shape and Motion Capture Using Multi-Modal Data (2020, background): “Follow-up works achieved better performance by utilizing priors and context [25], high-level knowledge [39], a feedback loop [26, 27], or intermediate dense guidance map supervision [46].”
Spatial Attention Deep Net with Partial PSO for Hierarchical Hybrid Hand Pose Estimation (2016, methodology): “We compare our work with 4 state-of-the-arts methods: Hierarchical Sampling Optimization(HSO) [19], Sharp et al. [3], HandsDeep [18], FeedLoop [4] on three datasets, see Fig.”
Deep Kinematic Pose Regression (2016, methodology): “In this work, we propose to directly incorporate the articulated object model into the deep neutral network learning, which is the dominant approach for object pose estimation nowadays, for hand [32,29,21,22,41,8] or human body[33,35,20,10,17,18,31,30,40].”