DoubleFusion: Real-Time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor

By Tao Yu, Zerong Zheng, Kaiwen Guo, Jianhui Zhao, Qionghai Dai, Hao Li, Gerard Pons-Moll, Yebin Liu. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

298 citations recorded by the source corpus.

We propose DoubleFusion, a new real-time system that combines volumetric dynamic reconstruction with data-driven template fitting to simultaneously reconstruct detailed geometry, non-rigid motion and the inner human body shape from a single depth camera. One of the key contributions of this method is a double layer representation consisting of a complete parametric body shape inside, and a gradually fused outer surface layer. A pre-defined node graph on the body surface parameterizes the non-rigid deformations near the body, and a free-form dynamically changing graph parameterizes the outer surface layer far from the body, which allows more general reconstruction. We further propose a joint motion tracking method based on the double layer representation to enable robust and fast motion tracking performance. Moreover, the inner body shape is optimized online and forced to fit inside the outer surface layer. Overall, our method enables increasingly denoised, detailed and complete surface reconstructions, fast motion tracking performance and plausible inner body shape reconstruction in real-time. In particular, experiments show improved fast motion tracking and loop closure performance on more challenging scenarios.

View Paper: https://www.semanticscholar.org/paper/fd24b1b5f8b77c6373e654ce9ec8ffc680336291

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Implicit Functions in Feature Space for 3D Shape Reconstruction and Completion (2020, methodology): “Such implicit representation has been combined with the SMPL [41] body model to significantly increase tracking robustness and accuracy [78].”
Function4D: Real-time Human Volumetric Capture from Very Sparse Consumer RGBD Sensors (2021, background): “…refer to Fig.5. icFusion, the following works incorporated killing/sobolev constraints [39, 40], articulated priors [21], skeleton tracking [54], parametric body models [55], sparse inertial measurements [56], data-driven prior [42] or learned correspondences [3] into the non-rigid fusion pipeline.”
Deep Learning-based Human Pose Estimation: A Survey (2020, background): “Yu et al. [275], Xiong et al. [261], Kadkhodamohammadi et al. [94], and Zhi et al. [305] utilized depth images to estimate 3D human pose.”
ARCH: Animatable Reconstruction of Clothed Humans (2020, background): “There are multiple attempts to solve this task with video inputs [2, 3, 37, 1, 52], RGB-D data [53, 56] and in multi-view settings [5, 13, 14, 45, 46, 47, 48, 6].”
DeepHuman: 3D Human Reconstruction From a Single Image (2019, methodology): “We leverage the state-of-theart DoubleFusion [63] technique for real-time human mesh reconstruction and propose a capture pipeline for fast and efficient capture of outer geometry of human bodies wearing casual clothes with medium-level surface detail and texture.”
Combining Implicit Function Learning and Parametric Models for 3D Human Reconstruction (2020, methodology): “Such free-form representation has been combined with SMPL [27] to increase robustness and tracking [56].”