Pixel-aligned Volumetric Avatars

By Amit Raj, Michael Zollhoefer, Tomas Simon, Jason M. Saragih, Shunsuke Saito, James Hays, Stephen Lombardi. 2021

52 citations recorded by the source corpus.

Acquisition and rendering of photo-realistic human heads is a highly challenging research problem of particular importance for virtual telepresence. Currently, the highest quality is achieved by volumetric approaches trained in a person-specific manner on multi-view data. These models better represent fine structure, such as hair, compared to simpler mesh-based models. Volumetric models typically employ a global code to represent facial expressions, such that they can be driven by a small set of animation parameters. While such architectures achieve impressive rendering quality, they can not easily be extended to the multi-identity setting. In this paper, we devise a novel approach for predicting volumetric avatars of the human head given just a small number of inputs. We enable generalization across identities by a novel parameterization that combines neural radiance fields with local, pixel-aligned features extracted directly from the inputs, thus side-stepping the need for very deep or complex networks. Our approach is trained in an end-to-end manner solely based on a photometric re-rendering loss without requiring explicit 3D supervision. We demonstrate that our approach outperforms the existing state of the art in terms of quality and is able to generate faithful facial expressions in a multi-identity setting.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

1. Bo Peng
2. Hujun Bao
3. Ligang Liu
4. Juyong Zhang

In Computer Graphics and Computer-Aided Design

1. Bo Peng
2. Hujun Bao
3. Ligang Liu
4. Juyong Zhang
5. Yang Hong
6. Haiyao Xiao

In Computer Vision and Pattern Recognition

1. Bo Peng
2. Hujun Bao
3. Ligang Liu
4. Yongjin Liu
5. Juyong Zhang
6. Sen Liang
7. Yang Hong
8. Haiyao Xiao

In 3D Vision & Pose

1. Bo Peng
2. Hujun Bao
3. Yongjin Liu
4. Juyong Zhang
5. Sen Liang
6. Keyu Chen
7. Yudong Guo

Citations in the corpus

• Neural 3D Video Synthesis from Multi-view Video (2021, unknown): “[47] Amit Raj, Michael Zollhöfer, Tomas Simon, Jason M.”
• AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, background): “[37] adopt pixel-aligned features [38] in NeRF to generalize to unseen identities at test time.”
• HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, methodology): “Benefiting from these advantages, NeRF has been widely used in many fields, such as 3D modeling [53, 58], human face/body digitization [12, 17, 42, 43, 48, 55], generating 4D free-view video [31,39], etc.”