Neural Body: Implicit Neural Representations with Structured Latent Codes for Novel View Synthesis of Dynamic Humans

By Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, Xiaowei Zhou. Computer Vision and Pattern Recognition, 2020

874 citations recorded by the source corpus.

This paper addresses the challenge of novel view synthesis for a human performer from a very sparse set of camera views. Some recent works have shown that learning implicit neural representations of 3D scenes achieves remarkable view synthesis quality given dense input views. However, the representation learning will be ill-posed if the views are highly sparse. To solve this ill-posed problem, our key idea is to integrate observations over video frames. To this end, we propose Neural Body, a new human body representation which assumes that the learned neural representations at different frames share the same set of latent codes anchored to a deformable mesh, so that the observations across frames can be naturally integrated. The deformable mesh also provides geometric guidance for the network to learn 3D representations more efficiently. To evaluate our approach, we create a multi-view dataset named ZJU-MoCap that captures performers with complex motions. Experiments on ZJU-MoCap show that our approach outperforms prior works by a large margin in terms of novel view synthesis quality. We also demonstrate the capability of our approach to reconstruct a moving person from a monocular video on the People-Snapshot dataset.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/af8faec7c0b8f4b2a28d42a86e0e7d499016c560

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, neutral): “Another line of research focuses on predeﬁned template models or skeletons to support actions like re-posing or re-rendering within speciﬁc categories [30, 33].”
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, background): “[33] integrate observations across video frames to enable novel view synthesis for human body from a sparse multi-view video.”
Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video (2020, background): “Others focus on moving human bodies [85, 58, 75] or more general objects [61, 37, 86, 57, 11, 36].”
IBRNet: Learning Multi-View Image-Based Rendering (2021, background): “While NeRF opens up many new research opportunities [30, 36, 43, 45, 55, 58], it must be optimized for each new scene, taking hours or days to converge.”
UniSim: A Neural Closed-Loop Sensor Simulator (2023, background): “We also do not model animation ( e.g ., turn signals, traffic lights) or actor deformation ( e.g . walking) and hope to incorporate ideas from works [57,59,90,99] that tackle this.”
Advances in Neural Rendering (2021, background): “Similarly to Neural Body [PZX ∗ 21], they require a sparse set of videos from synchronized and calibrated cameras.”
A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose (2021, methodology): “While NeuralBody [49] can also learn photo-realistic human models from monocular images, their model anchors its representation on the SMPL 3D”
NeuMan: Neural Human Radiance Field from a Single Video (2022, positive): “Generally, motion capture data [33,11] is captured with a static multi-cameras system in a Fig.”
HexPlane: A Fast Representation for Dynamic Scenes (2023, background): “Representing dynamic scenes by neural radiance ﬁelds is an essential extension of NeRF, enabling numerous real-world applications [30,51,56,70,83,89,96].”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, methodology): “This approach has been applied for deforming faces [12, 14, 45], moving human bodies [43, 53, 56], and objects [10,29,42,44,59].”
NeuS2: Fast Learning of Neural Implicit Surfaces for Multi-view Reconstruction (2022, background): “Some works in human performance modeling [54, 31, 43, 7, 63, 38, 19, 44, 24, 60] can model large movements by introducing a deformable template as a prior.”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, methodology): “Benefiting from these advantages, NeRF has been widely used in many fields, such as 3D modeling [53, 58], human face/body digitization [12, 17, 42, 43, 48, 55], generating 4D free-view video [31,39], etc.”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, background): “Neural Body applied NeRF volume rendering to rendering humans with moving poses from videos.”