State of the Art on Monocular 3D Face Reconstruction, Tracking, and Applications

By Michael Zollhöfer, Justus Thies, Pablo Garrido, Derek Bradley, Thabo Beeler, Patrick Pérez, Marc Stamminger, Matthias Nießner, Christian Theobalt. Computer graphics forum (Print), 2018

344 citations recorded by the source corpus.

The computer graphics and vision communities have dedicated long standing efforts in building computerized tools for reconstructing, tracking, and analyzing human faces based on visual input. Over the past years rapid progress has been made, which led to novel and powerful algorithms that obtain impressive results even in the very challenging case of reconstruction from a single RGB or RGB‐D camera. The range of applications is vast and steadily growing as these technologies are further improving in speed, accuracy, and ease of use.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

View Paper: https://www.semanticscholar.org/paper/fda48e021f7c1445406fa7db32a443726e0ef8d3

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Deep video portraits (2018, methodology): “For a comprehensive overview of current methods we refer to a recent state-of-the-art report on monocular 3D face reconstruction, tracking and applications [Zollhöfer et al. 2018].”
Nerfies: Deformable Neural Radiance Fields (2020, methodology): “Although some methods show impressive results in monocular face reconstruction from color and RGBD cameras [63], such models often lack detail (e.”
Expressive Body Capture: 3D Hands, Face, and Body From a Single Image (2019, methodology): “There are many methods that estimate 3D faces from images or RGB-D [81] as well as methods that estimate hands from such data [80].”
On the Detection of Digital Face Manipulation (2019, methodology): “3D face reconstruction and animation methods [17, 32, 48, 64] are widely used for expression swap, such as Face2Face [47].”
3D Morphable Face Models—Past, Present, and Future (2019, neutral): “There is a recent state of the art report on monocular 3D reconstruction, tracking and applications [Zollhoefer et al. 2018].”
Exploiting Visual Artifacts to Expose Deepfakes and Face Manipulations (2019, methodology): “A state-of-the-art method that uses similar methods can be found in [42].”
HeadNeRF: A Realtime NeRF-based Parametric Head Model (2021, background): “The parametric face/head model, which encodes the human face/head in low-dimensional space, is a hot research topic in computer vision and computer graphics and widely used in many applications like identity recognition [34, 52], face analysis [11, 64] and film/game production [47], etc.”
First Order Motion Model for Image Animation (2020, background): “Traditional approaches for image animation and video re-targeting [6, 33, 13] were designed for specific domains such as faces [45, 42], human silhouettes [8, 37, 27] or gestures [31] and required a strong prior of the animated object.”
Cross-Domain and Disentangled Face Manipulation With 3D Guidance (2021, background): “manipulation applications [33], [34], [35], [36], [37], but they only capture the coarse geometry due to the mostly linear statistical model.”
Learning an animatable detailed 3D face model from in-the-wild images (2020, background): “Since then, 3D face reconstruction methods have rapidly advanced (for a comprehensive overview see [Morales et al. 2021; Zollhöfer et al. 2018]) enabling applications such as 3D avatar creation for VR/AR [Hu et al. 2017], video editing [Kim et al. 2018a; Thies et al. 2016], image synthesis [Ghosh…”
Attributing Fake Images to GANs: Learning and Analyzing GAN Fingerprints (2018, background): “Visual contents can now be easily created and edited without leaving obvious perceptual traces [72].”
EMOCA: Emotion Driven Monocular Face Capture and Animation (2022, background): “Monocular face reconstruction: Reconstructing 3D face shape from images has been studied extensively for more than two decades [22, 99].”
StyleRig: Rigging StyleGAN for 3D Control Over Portrait Images (2020, background): “For an in-depth overview of parametric face models and their possible applications we refer to the recent survey papers [10, 39].”
X2Face: A network for controlling face generation by using images, audio, and pose codes (2018, neutral): “[46] for a survey.”
Neural Voice Puppetry: Audio-driven Facial Reenactment (2019, background): “In the recent years, facial reenactment has witnessed a growing interested from the research community [28, 36].”