A-NeRF: Articulated Neural Radiance Fields for Learning Human Shape, Appearance, and Pose

By Shih-Yang Su, F. Richard Yu, Michael Zollhoefer, Helge Rhodin. Neural Information Processing Systems, 2021

311 citations recorded by the source corpus.

While deep learning reshaped the classical motion capture pipeline with feed-forward networks, generative models are required to recover fine alignment via iterative refinement. Unfortunately, the existing models are usually hand-crafted or learned in controlled conditions, only applicable to limited domains. We propose a method to learn a generative neural body model from unlabelled monocular videos by extending Neural Radiance Fields (NeRFs). We equip them with a skeleton to apply to time-varying and articulated motion. A key insight is that implicit models require the inverse of the forward kinematics used in explicit surface models. Our reparameterization defines spatial latent variables relative to the pose of body parts and thereby overcomes ill-posed inverse operations with an overparameterization. This enables learning volumetric body shape and appearance from scratch while jointly refining the articulated pose; all without ground truth labels for appearance, pose, or 3D shape on the input videos. When used for novel-view-synthesis and motion capture, our neural model improves accuracy on diverse datasets. Project website: https://lemonatsu.github.io/anerf/ .

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition, Control and Systems Engineering

View Paper: https://www.semanticscholar.org/paper/74bb19d1ce2ec9fb9605545a750e68281a067e63

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Block-NeRF: Scalable Large Scene Neural View Synthesis (2022, methodology): “Our model also incorporates a learned camera pose refinement which has been explored in previous works [32, 58, 66, 70, 71].”
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels (2022, background): “A series of articulated NeRFmethods [Noguchi et al. 2021; Su et al. 2021; Weng et al. 2022; Xu et al. 2021] are also proposed to represent human body motions.”
HumanNeRF: Free-viewpoint Rendering of Moving People from Monocular Video (2022, background): “Su et al. [60] use an over-parameterized NeRF to rigidly transform NeRF features for refining body pose and thus final rendering.”