HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation

By Jiefeng Li, Chao Xu, Zhicun Chen, Siyuan Bian, Lixin Yang, Cewu Lu. Computer Vision and Pattern Recognition, 2020

517 citations recorded by the source corpus.

Model-based 3D pose and shape estimation methods reconstruct a full 3D mesh for the human body by estimating several parameters. However, learning the abstract parameters is a highly non-linear process and suffers from image-model misalignment, leading to mediocre model performance. In contrast, 3D keypoint estimation methods combine deep CNN network with the volumetric representation to achieve pixel-level localization accuracy but may predict unrealistic body structure. In this paper, we address the above issues by bridging the gap between body mesh estimation and 3D keypoint estimation. We propose a novel hybrid inverse kinematics solution (HybrIK). HybrIK directly transforms accurate 3D joints to relative body-part rotations for 3D body mesh reconstruction, via the twistand-swing decomposition. The swing rotation is analytically solved with 3D joints, and the twist rotation is derived from the visual cues through the neural network. We show that HybrIK preserves both the accuracy of 3D pose and the realistic body structure of the parametric human model, leading to a pixel-aligned 3D body mesh and a more accurate 3D pose than the pure 3D keypoint estimation methods. Without bells and whistles, the proposed method surpasses the state-of-the-art methods by a large margin on various 3D human pose and shape benchmarks. As an illustrative example, HybrIK outperforms all the previous methods by 13.2 mm MPJPE and 21.9 mm PVE on 3DPW dataset. Our code is available at https://github.com/Jeff-sjtu/HybrIK.

Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/e000b443c1999ea941d36108fbd6463d8f621a42

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

MotionBERT: A Unified Perspective on Learning Human Motion Representations (2022, methodology): “Note that most previous works [23,38,42,44,50,54,68,107] use more datasets other than COCO [57] during training, such as LSP [37], MPI-INF-3DHP [74], etc ., while we do not.”
CLIFF: Carrying Location Information in Full Frames into Human Pose and Shape Estimation (2022, unknown): “Both top-down [23,25,21,27] and bottom-up [49] approaches can do the job.”
Human Pose Regression with Residual Log-likelihood Estimation (2021, unknown): “The 3D heatmap representation is followed by several works [54, 35, 7, 69, 59, 28].”
Perpetual Humanoid Control for Real-time Simulated Avatars (2023, methodology): “We use HybrIK[17] to estimate the joint rotations ˜ θ t and uses MeTRAbs [39] for global 3D keypoints ˜ p t .”
Deep Learning-based Human Pose Estimation: A Survey (2020, background): “, [6, 11, 102, 116, 168, 288, 290, 314], because it is compatible with existing rendering engines.”