Animatable Neural Radiance Fields for Modeling Dynamic Human Bodies

By Sida Peng, Junting Dong, Qianqian Wang, Qing Shuai, Xiaowei Zhou, Hujun Bao. IEEE International Conference on Computer Vision, 2021

474 citations recorded by the source corpus.

This paper addresses the challenge of reconstructing an animatable human model from a multi-view video. Some recent works have proposed to decompose a non-rigidly deforming scene into a canonical neural radiance field and a set of deformation fields that map observation-space points to the canonical space, thereby enabling them to learn the dynamic scene from images. However, they represent the deformation field as translational vector field or SE(3) field, which makes the optimization highly under-constrained. Moreover, these representations cannot be explicitly controlled by input motions. Instead, we introduce neural blend weight fields to produce the deformation fields. Based on the skeleton-driven deformation, blend weight fields are used with 3D human skeletons to generate observation-to-canonical and canonical-to-observation correspondences. Since 3D human skeletons are more observable, they can regularize the learning of deformation fields. Moreover, the learned blend weight fields can be combined with input skeletal motions to generate new deformation fields to animate the human model. Experiments show that our approach significantly outperforms recent human synthesis methods. The code and supplementary materials are available at https://zju3dv.github.io/animatable_nerf/.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/35e940ace815548f620709d9c1803da34c581e86

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields (2021, unknown): “NeRF has inspired many subsequent works, which extend its neural volumetric scene representation to application domains including dynamic and deformable scenes [24], avatar animation [10, 25], and even phototourism [19].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “State of the art models from top tier conferences in 2021/2022 such as Animatable NeRF [144] (May 2021), DoubleField [143] (Jun 2021), HumanNeRF [141] (Jan 2022), Zheng et al. [142] (Mar 2022) also innovated on this topic.”
A Survey on 3D Gaussian Splatting (2024, unknown): “• Benchmarking Algorithms: For performance comparison, we involve four recent 3D GS based algorithms [95, 149, 254, 279] and six typical SLAM methods [192, 311].”
NeuMan: Neural Human Radiance Field from a Single Video (2022, unknown): “Recent efforts also focus on animation of these radiance field models [19,33,32,12,40] of human, with the aid of large controlled datasets, further extending the application domain of radiance-field-based modeling to enable augmented reality experiences.”
GaussianAvatars: Photorealistic Head Avatars with Rigged 3D Gaussians (2023, background): “Peng et al. [34] deform points with the skeleton of SMPL and neural blending weights.”
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video (2022, methodology): “Other than free-viewpoint rendering, there is another related active research field that focus on human motion retargeting either in 2D [2, 7, 35, 43, 54, 67, 68] or 3D [19, 20, 25, 32, 49, 53, 69, 74].”