HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video

By Chung-Yi Weng, Brian Curless, Pratul P. Srinivasan, Jonathan T. Barron, Ira Kemelmacher-Shlizerman. Computer Vision and Pattern Recognition, 2022

654 citations recorded by the source corpus.

We introduce a free-viewpoint rendering method - HumanNeRF - that works on a given monocular video of a human performing complex body motions, e.g. a video from YouTube. Our method enables pausing the video at any frame and rendering the subject from arbitrary new camera viewpoints or even a full 360-degree camera path for that particular frame and body pose. This task is particularly challenging, as it requires synthesizing photorealistic details of the body, as seen from various camera angles that may not exist in the input video, as well as synthesizing fine details such as cloth folds and facial appearance. Our method optimizes for a volumetric representation of the person in a canonical T-pose, in concert with a motion field that maps the estimated canonical representation to every frame of the video via backward warps. The motion field is decomposed into skeletal rigid and non-rigid motions, produced by deep networks. We show significant performance improvements over prior work, and compelling examples of free-viewpoint renderings from monocular video of moving humans in challenging uncontrolled capture scenarios.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/f763a59644e27a2215095943224f2564e670a504

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Humans in 4D: Reconstructing and Tracking Humans with Transformers (2023, background): “0 is such a large pre-trained model which could potentially be useful not just in computer vision, but also in robotics [52, 59, 70], computer graphics [73], biomechanics, and other fields where analysis of the human figure and its movement from images or videos is needed.”
Advances in Neural Rendering (2021, background): “Another work called HumanNeRF [WCS ∗ 22] shows how to train a neural radiance ﬁeld for a speciﬁc actor based on monocular input data, using a skeleton-driven motion ﬁeld which is reﬁned by a general non-rigid motion ﬁeld.”
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, background): “Additionally, other notable works, namely [4, 14, 38, 41], have put forth effective methodologies for acquiring precise dynamic human body motion information, building upon the foundation provided by the SMPL [18] model.”
Tensor4D: Efficient Neural 4D Decomposition for High-Fidelity Dynamic Reconstruction and Rendering (2022, methodology): “Using parametric body templates as the semantic prior, methods like Neural Body [38] and HumanNeRF [60] enable photo-realistic novel view synthesis of complex human performance.”
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels (2022, background): “A series of articulated NeRFmethods [Noguchi et al. 2021; Su et al. 2021; Weng et al. 2022; Xu et al. 2021] are also proposed to represent human body motions.”
NeuMan: Neural Human Radiance Field from a Single Video (2022, positive): “Similar to HumanNeRF [47], we also use LPIPS [51] as an additional loss term L lpips by sampling a 32 × 32 patch.”
Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis (2023, neutral): “(d) Template guided methods [13, 20, 42], which model dynamic scenes in restricted environments where the motion can be modelled by a predefined template e.”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, neutral): “…becomes a popular 3D representation for photo-realistic novel-view synthesis in recent years [35], many variants are proposed to improve quality [2–4, 24, 45, 47–49, 57], increase speed [8, 9, 11, 14, 20, 32, 36, 37, 40, 43, 46, 58, 60], extend to dynamic scenes [7, 15, 18, 25, 27, 50], and so on.”