Deferred neural rendering

By Justus Thies, Michael Zollhöfer, Matthias Nießner. ACM Transactions on Graphics, 2019

818 citations recorded by the source corpus.

The modern computer graphics pipeline can synthesize images at remarkable visual quality; however, it requires well-defined, high-quality 3D content as input. In this work, we explore the use of imperfect 3D content, for instance, obtained from photo-metric reconstructions with noisy and incomplete surface geometry, while still aiming to produce photo-realistic (re-)renderings. To address this challenging problem, we introduce Deferred Neural Rendering, a new paradigm for image synthesis that combines the traditional graphics pipeline with learnable components. Specifically, we propose Neural Textures, which are learned feature maps that are trained as part of the scene capture process. Similar to traditional textures, neural textures are stored as maps on top of 3D mesh proxies; however, the high-dimensional feature maps contain significantly more information, which can be interpreted by our new deferred neural rendering pipeline. Both neural textures and deferred neural renderer are trained end-to-end, enabling us to synthesize photo-realistic images even when the original 3D content was imperfect. In contrast to traditional, black-box 2D generative neural networks, our 3D representation gives us explicit control over the generated output, and allows for a wide range of application domains. For instance, we can synthesize temporally-consistent video re-renderings of recorded 3D scenes as our representation is inherently embedded in 3D space. This way, neural textures can be utilized to coherently re-render or manipulate existing video content in both static and dynamic environments at real-time rates. We show the effectiveness of our approach in several experiments on novel view synthesis, scene editing, and facial reenactment, and compare to state-of-the-art approaches that leverage the standard graphics pipeline as well as conventional generative neural networks.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/1795a62eb1bfd952c73acb18ca81f174b0042cae

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

WildDeepfake: A Challenging Real-World Dataset for Deepfake Detection (2020, methodology): “…[23] dataset has 1,000 real videos collected from YouTube, based on which 1,000 deepfake videos were generated by applying each of the 4 face modification techniques: Deepfake [2], Face2Face [29], Faceswap [4] and Neural Texture [28] (eg. overall 4,000 face modification videos were created).”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, methodology): “Further, by integrating a neural rendering pipeline [20, 40, 41, 48, 61, 79, 80, 82, 83], our model scales to more complex, real-world data.”
Exploring Temporal Coherence for More General Video Face Forgery Detection (2021, methodology): “The fake videos are manipulated by four methods: Face2Face(F2F) [49], FaceSwap(FS) [2], NeuralTexture(NT) [48], and Deepfake(DF) [1].”
pixelNeRF: Neural Radiance Fields from One or Few Images (2020, background): “ward photorealism for both wider ranges of novel views and sparser sets of input views, by using 3D representations based on neural networks [27, 23, 26, 38, 42, 7].”
DeepFakes and Beyond: A Survey of Face Manipulation and Fake Detection (2020, background): “Later on, the same authors presented in FaceForensics++ a new learning approach based on NeuralTextures [102].”
FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping (2019, background): “6 [37] Justus Thies, Michael Zollhöfer, and Matthias Nießner.”
Two-branch Recurrent Network for Isolating Deepfakes in Videos (2020, methodology): “manipulated frames using four methods: FaceSwap, DeepFakes, Face2Face [57], and NeuralTextures [56].”
Cross-Domain and Disentangled Face Manipulation With 3D Guidance (2021, background): “differentiable graphics pipelines [38], [39], [40], combining the traditional pipelines with learnable components [41],”
FaceForensics++: Learning to Detect Manipulated Facial Images (2019, methodology): “In particular, the benchmark is based on DeepFakes [1], Face2Face [56], FaceSwap [2] and NeuralTextures [54] as prominent representatives for facial manipulations at random compression level and size.”
HyperNeRF (2021, methodology): “…or illumination models into the “translation” process [Meka et al. 2019; Sun et al. 2019], or by using proxy geometries [Aliev et al. 2019; Thies et al. 2019] or conventional renderings [Fried et al. 2019; Kim et al. 2018; Martin-Brualla et al. 2018; Meshry et al. 2019] as a harness…”
End-to-End Reconstruction-Classification Learning for Face Forgery Detection (2022, background): “The recent years have witnessed the considerable progress of face forgery generation methods [2,4,11,18,20, 41, 42, 50, 53].”
DeeperForensics-1.0: A Large-Scale Dataset for Real-World Face Forgery Detection (2020, background): “, DeepFakes [2], Face2Face [36], FaceSwap [3], NeuralTextures [35])), and 1, 000 real videos from YouTube.”
Celeb-DF: A Large-Scale Challenging Dataset for DeepFake Forensics (2019, background): “Although in recent years there have been many sophisticated algorithms for generating realistic synthetic face videos [9, 13, 46, 51, 26, 47, 37, 20, 23, 10, 21, 50], most of these have not been in mainstream as open-source software tools that anyone can use.”