Real-time non-rigid reconstruction using an RGB-D camera

By Michael Zollhöfer, Matthias Nießner, Shahram Izadi, Christoph Rhemann, Christopher Zach, Matthew Fisher, Chenglei Wu, Andrew William Fitzgibbon, Charles T. Loop, Christian Theobalt and 1 more. ACM Transactions on Graphics, 2014

467 citations recorded by the source corpus.

Subfields: Aerospace Engineering, Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/9c22dd54aa7284b1ae32822f8b9d18fbf58f258a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

VNect (2017, background): “Even real-time tracking of general deforming objects [Zollhöfer et al. 2014] and template-free reconstruction [Dou et al.”
Dynamic FAUST: Registering Human Bodies in Motion (2017, background): “For example, there are many non-rigid tracking methods, either model-based [3, 7, 22, 25, 27, 34, 44, 49] or model-free [2, 16, 21, 24, 37, 50, 54].”
Co-fusion: Real-time segmentation, tracking and fusion of multiple objects (2017, background): “On the other hand template based techniques can also obtain high levels of realism but are limited by their need to add a preliminary step to capture the template [33] or are dedicated to tracking specific objects by their use of handcrafted or pre-trained models [27].”
Detailed, Accurate, Human Shape Estimation from Clothed 3D Scan Sequences (2017, background): “Other work is model free and estimates non-rigid 3D shape over time [10, 22, 25, 48].”
Nerfies: Deformable Neural Radiance Fields (2020, methodology): “In the vision community, these energies have been extensively used for the reconstruction and tracking of non-rigid scenes and objects [17, 37, 62] interpolation w/o elastic regularization”
Deepfakes generation and detection: state-of-the-art, open challenges, countermeasures, and way forward (2021, background): “[125, 126] presented the first real-time facial expression transfer method”
Learning to Reconstruct People in Clothing From a Single RGB Camera (2019, background): “from a low number of video cameras [60], or using one or more depth cameras, either specifically for the human body [9, 77, 84] or for general free-form surfaces [88, 51, 54, 37, 26, 70].”
DoubleFusion: Real-Time Capture of Human Performances with Inner Body Shapes from a Single Depth Sensor (2018, background): “As depth cameras are increasingly popular in the consumer space (iPhoneX, Google Tango, etc.), the recent trend focuses on using more and more practical setups like a single depth camera [45, 13, 3].”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, background): “Examples include 3D reconstruction for static scenes [49, 27] or dynamic objects from a single [45, 26] or RGB-D cameras [14, 5, 26, 45, 73, 79].”
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes (2020, background): “Most successful nonrigid reconstruction systems either require RGBD data as input [7, 18, 29, 48, 80, 86], or can only reconstruct sparse geometry [56, 65, 77, 83].”
Fusion4D (2016, methodology): “Our approach to the linear solver is akin to the approach taken by [Zollhöfer et al. 2014], but instead of implementing our solver in terms of Jf and J>f , we use terms J>J and J>f .”
DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time (2015, methodology): “Recently, [37] demonstrated an impressive real-time version of a similar technique, using GPU accelerated optimisations.”
Real-time expression transfer for facial reenactment (2015, methodology): “In previous work [Zollhöfer et al. 2014], the PCG solver is optimized for a sparse Jacobian and the entries of J are computed on-the-fly in each iteration.”
Face2Face: Real-Time Face Capture and Reenactment of RGB Videos (2016, methodology): “As suggested by [33, 29], we split up the multiplication of the old descent direction d with the system matrix JJ in the PCG solver into two successive matrixvector products.”
Holoportation: Virtual 3D Teleportation in Real-time (2016, background): “PatchMatch stereo alternates between random depth generation and propagation of depth based on the algorithm by [2], and has recently been extended to real-time performance by assuming fronto parallel windows and reducing the number of iterations of depth propagation [65].”