VolumeDeform: Real-Time Volumetric Non-rigid Reconstruction

By Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, Marc Stamminger. 2016

349 citations recorded by the source corpus.

We present a novel approach for the reconstruction of dynamic geometric shapes using a single hand-held consumer-grade RGB-D sensor at real-time rates. Our method builds up the scene model from scratch during the scanning process, thus it does not require a pre-defined shape template to start with. Geometry and motion are parameterized in a unified manner by a volumetric representation that encodes a distance field of the surface geometry as well as the non-rigid space deformation. Motion tracking is based on a set of extracted sparse color features in combination with a dense depth constraint. This enables accurate tracking and drastically reduces drift inherent to standard model-to-depth alignment. We cast finding the optimal deformation of space as a non-linear regularized variational optimization problem by enforcing local smoothness and proximity to the input constraints. The problem is tackled in real-time at the camera’s capture rate using a data-parallel flip-flop optimization strategy. Our results demonstrate robust tracking even for fast motion and scenes that lack geometric features.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Deformable Shape Completion with Graph Convolutional Autoencoders (2017, background): “The works of [36] and [21] have shown very impressive reconstructions, however they are limited to small motions between consecutive frames.”
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes (2020, methodology): “Most successful nonrigid reconstruction systems either require RGBD data as input [7, 18, 29, 48, 80, 86], or can only reconstruct sparse geometry [56, 65, 77, 83].”
DeepHuman: 3D Human Reconstruction From a Single Image (2019, background): “Many of the following approaches have tried to improve the robustness by adding color features [23], shading constraints [19] and articulated prior [75] and dealing with topology changes [56, 57].”
DoubleFusion: Real-Time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor (2018, methodology): “In particular, by combining non-rigid surface tracking and volumetric depth integration, DynamicFusion like approaches [28, 15, 14, 34] allow real-time dynamic scene reconstruction using a single depth camera without the requirement of pre-scanned model templates.”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, neutral): “2015), VolumeDeform (Innmann et al., 2016), and Fusion4D (Dou et al.”
MaskFusion: Real-Time Recognition, Tracking and Reconstruction of Multiple Moving Objects (2018, background): “Although some first steps have been taken towards non-rigid and dynamic scene reconstruction, with exciting results in reconstruction of a single non-rigid object [12, 20, 32, 53] or multiple moving rigid objects [39], designing an accurate and robust SLAM system that can deal with arbitrary dynamic”
Video Based Reconstruction of 3D People Models (2018, background): “While general, such template-free approaches [45, 31, 60] are limited to slow and careful motions.”
ClothCap (2017, background): “Other works also use a free-form surface (Collet et al. 2015; Innmann et al. 2016; Newcombe et al. 2015; Wang et al. 2016) but can not separate the clothing from the person and can not retarget it.”
Learning the Depths of Moving People by Watching Frozen People (2019, methodology): “RGBD data has been widely used for 3D modeling of dynamic scenes [25, 55, 48, 5, 14], but only a few methods attempt to estimate depth from a monocular camera.”
Neural volumes (2019, background): “Zollhöfer et al. [2014] and Innmann et al. [2016] also deform a 3D template surface with an as-rigid-as-possible regularizer.”
VNect (2017, methodology): “2014] and template-free reconstruction [Dou et al. 2016; Innmann et al. 2016; Newcombe et al. 2015; Orts-Escolano et al. 2016] has been demonstrated.”
Multi-Garment Net: Learning to Dress 3D People From Images (2019, background): “Without a pre-scanned template, fusion [30, 29, 55, 43, 58] trackers incrementally fuse geometry and appearance [66] to build the template on the fly.”
Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video (2020, methodology): “Such sensors can been used for 3D reconstruction and completion of rigid environments [51] and non-rigid objects [98, 50, 28, 71].”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, background): “Examples include 3D reconstruction for static scenes [49, 27] or dynamic objects from a single [45, 26] or RGB-D cameras [14, 5, 26, 45, 73, 79].”
Deferred neural rendering (2019, background): “A major benefit of this line of work is that the reconstructed 3D content can be visualized by standard rendering techniques, and the approach directly generalizes to 4D capture as long as non-rigid surfaces can be reliably tracked [Dou et al. 2016; Innmann et al. 2016; Newcombe et al. 2015].”
Learning to Reconstruct People in Clothing From a Single RGB Camera (2019, background): “from a low number of video cameras [60], or using one or more depth cameras, either specifically for the human body [9, 77, 84] or for general free-form surfaces [88, 51, 54, 37, 26, 70].”