V2V-PoseNet: Voxel-to-Voxel Prediction Network for Accurate 3D Hand and Human Pose Estimation from a Single Depth Map

By Gyeongsik Moon, Ju Yong Chang, Kyoung Mu Lee. 2017

470 citations recorded by the source corpus.

Most of the existing deep learning-based methods for 3D hand and human pose estimation from a single depth map are based on a common framework that takes a 2D depth map and directly regresses the 3D coordinates of keypoints, such as hand or human body joints, via 2D convolutional neural networks (CNNs). The first weakness of this approach is the presence of perspective distortion in the 2D depth map. While the depth map is intrinsically 3D data, many previous methods treat depth maps as 2D images that can distort the shape of the actual object through projection from 3D to 2D space. This compels the network to perform perspective distortion-invariant estimation. The second weakness of the conventional approach is that directly regressing 3D coordinates from a 2D image is a highly nonlinear mapping, which causes difficulty in the learning procedure. To overcome these weaknesses, we firstly cast the 3D hand and human pose estimation problem from a single depth map into a voxel-to-voxel prediction that uses a 3D voxelized grid and estimates the per-voxel likelihood for each keypoint. We design our model as a 3D CNN that provides accurate estimates while running in real-time. Our system outperforms previous methods in almost all publicly available 3D hand and human pose estimation datasets and placed first in the HANDS 2017 frame-based 3D hand pose estimation challenge. The code is available in1.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Human-Computer Interaction

Citations in the corpus

Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
I2L-MeshNet: Image-to-Lixel Prediction Network for Accurate 3D Human Pose and Mesh Estimation from a Single RGB Image (2020, methodology): “The above limitations can make training harder, and as a result, reduce the test accuracy as addressed in [32,45].”
Deep Learning for Image Super-Resolution: A Survey (2019, background): “Depth maps record the distance between the viewpoint and the objects in the scene, and the depth information plays important roles in many tasks such as pose estimation [150], [151], [152], semantic segmentation [153], [154], etc.”
Depth-Based 3D Hand Pose Estimation: From Current Achievements to Future Goals (2017, methodology): “V2V-PoseNet [24] uses residual blocks as main building blocks.”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, background): “While a significant amount of research has focused on predicting the pose of hands [27, 54, 56, 58, 70, 93, 94, 98] or objects [5, 48, 59, 78, 83, 90] in isolation, joint understanding of handobject interactions has received far less attention.”
Monocular Real-Time Hand Shape and Motion Capture Using Multi-Modal Data (2020, background): “Some works leveraged other representations, such as point clouds [8, 11, 4, 18] and 3D voxels [16, 22, 9], which can be retrieved from depth images.”
ContactPose: A Dataset of Grasps with Object Contact and Hand Pose (2020, methodology): “ of the participant performing 7 known hand gestures (shown in the supplementary material). Along with 3D joints, this data can potentially enable tting of the MANO hand mesh model [50] to each grasp [41]. In this paper, we use meshes t to 3D joints (Figure4, see supp. mat. for details) for some of ”
Reconstructing Hand-Object Interactions in the Wild (2020, background): “Many previous works on hand pose estimation directly predict 3D joint locations from either depth [50, 53, 55, 59, 64, 65, 35] or RGB [45, 36, 4, 63, 68] images.”
Learnable Triangulation of Human Pose (2019, methodology): “Aggregated volumetric maps are then fed into a learnable volumetric convolutional neural network u (with weights denoted by ν), with architecture similar to V2V [14], producing the interpretable 3D-heatmaps of the output joints:”
Automated distance estimation for wildlife camera trapping (2022, background): “…has a wide variety of applications, for example, augmented reality applications (Woo et al., 2011), real-time 3D human pose and movement estimation (Moon et al., 2017), navigation of autonomous vehicles (Geiger et al., 2013) or robots, 3D photography (Kopf et al., 2020), 3D scene reconstruction…”
InterHand2.6M: A Dataset and Baseline for 3D Interacting Hand Pose Estimation from a Single RGB Image (2020, methodology): “nt. Ge et al. [5] extended this method by estimating multi-view 2D heatmaps. Guo et al. [7] proposed a region ensemble network to estimate the 3D coordinates of hand keypoints accurately. Moon et al. [15] designed a 3D CNN model that takes voxel input and outputs a 3D heatmap for each keypoint. Wan ”
Disentangling Latent Hands for Image Synthesis and Pose Estimation (2018, methodology): “State-of-the-art methods use a convolutional neural network (CNN) architecture, with the majority of works treating the depth input as 2D pixels, though a few more recent approaches treat depth inputs as a set of 3D points and or voxels [7, 5, 15].”
Knowledge As Priors: Cross-Modal Knowledge Generalization for Datasets Without Superior Knowledge (2020, background): “Estimating 3D hand poses from depth maps has made great progress in the past few years [10, 11, 26, 32, 43].”