FrankMocap: Fast Monocular 3D Hand and Body Motion Capture by Regression and Integration
By Yu Rong, Takaaki Shiratori, Hanbyul Joo. 2020
121 citations recorded by the source corpus.
Although the essential nuance of human motion is often conveyed as a combination of body movements and hand gestures, the existing monocular motion capture approaches mostly focus on either body motion capture only ignoring hand parts or hand motion capture only without considering body motion. In this paper, we present FrankMocap, a motion capture system that can estimate both 3D hand and body motion from in-the-wild monocular inputs with faster speed (9.5 fps) and better accuracy than previous work. Our method works in near real-time (9.5 fps) and produces 3D body and hand motion capture outputs as a unified parametric model structure. Our method aims to capture 3D body and hand motion simultaneously from challenging in-the-wild monocular videos. To construct FrankMocap, we build the state-of-the-art monocular 3D "hand" motion capture method by taking the hand part of the whole body parametric model (SMPL-X). Our 3D hand motion capture output can be efficiently integrated to monocular body motion capture output, producing whole body motion results in a unified parrametric model structure. We demonstrate the state-of-the-art performance of our hand motion capture system in public benchmarks, and show the high quality of our whole body motion capture result in various challenging real-world scenes, including a live demo scenario.
Subfields: Computer Vision and Pattern Recognition, Control and Systems Engineering, 3D Vision & Pose
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Jitendra Malik
- Cordelia Schmid
- Andrew Zisserman
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Andrea Vedaldi
In Control and Systems Engineering
- Angjoo Kanazawa
- Cordelia Schmid
- Gul Varol
- Jitendra Malik
- Ivan Laptev
- Yana Hasson
- Ilija Radosavovic
- Zhe Cao
In 3D Vision & Pose
- Angjoo Kanazawa
- Cordelia Schmid
- Ivan Laptev
- Yana Hasson
- Jitendra Malik
- Ilija Radosavovic
- Gul Varol
- Simon Stent
Citations in the corpus
- Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “One highly related direction is to model and estimate 3D hand joints [5,44,45,53,60,63,66,72,78,79,80,85,3] and mesh [26,27,51,55,75,83,46], object pose [14,28,29,35,43,57,64,71,76,81], or both [6,17,18,19,69].”
- Reconstructing Hand-Object Interactions in the Wild (2020, methodology): “We use a learning-based method [38] to obtain the initial hand pose estimation and further improve the result by imposing constraints on 2D hand keypoints and 3D hand-object contact priors.”
- Towards Unconstrained Joint Hand-Object Reconstruction From RGB Videos (2021, methodology): “We employ the recent publicly available hand pose estimator FrankMocap [54] to estimate the initial hand articulated poses, as well as the hand location and scale in pixel space.”