State of the Art on Neural Rendering

By Ayush Kumar Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Kalyan Sunkavalli, Ricardo Martin-Brualla, Tomas Simon, Jason M. Saragih, Matthias Nießner and 9 more. Computer graphics forum (Print), 2020

534 citations recorded by the source corpus.

Efficient rendering of photo‐realistic virtual worlds is a long standing effort of computer graphics. Modern graphics techniques have succeeded in synthesizing photo‐realistic images from hand‐crafted scene representations. However, the automatic generation of shape, materials, lighting, and other aspects of scenes remains a challenging problem that, if solved, would make photo‐realistic computer graphics more widely accessible. Concurrently, progress in computer vision and machine learning have given rise to a new approach to image synthesis and editing, namely deep generative models. Neural rendering is a new and rapidly emerging field that combines generative machine learning techniques with physical knowledge from computer graphics, e.g., by the integration of differentiable rendering into network training. With a plethora of applications in computer graphics and vision, neural rendering is poised to become a new area in the graphics community, yet no survey of this emerging field exists. This state‐of‐the‐art report summarizes the recent trends and applications of neural rendering. We focus on approaches that combine classic computer graphics techniques with deep generative models to obtain controllable and photorealistic outputs. Starting with an overview of the underlying computer graphics and machine learning concepts, we discuss critical aspects of neural rendering approaches. Specifically, our emphasis is on the type of control, i.e., how the control is provided, which parts of the pipeline are learned, explicit vs. implicit control, generalization, and stochastic vs. deterministic synthesis. The second half of this state‐of‐the‐art report is focused on the many important use cases for the described algorithms such as novel view synthesis, semantic photo manipulation, facial and body reenactment, relighting, free‐viewpoint video, and the creation of photo‐realistic avatars for virtual and augmented reality telepresence. Finally, we conclude with a discussion of the social implications of such technology and investigate open research problems.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/e3d5ee10d0489c768e943546038e3f53e7697349

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

StyleFlow: Attribute-conditioned Exploration of StyleGAN-Generated Images using Conditional Continuous Normalizing Flows (2020, methodology): “We refer the reader to a recent state of the art report (Tewari et al. 2020b) that summarizes recent techniques.”
Learning Physical Graph Representations from Visual Scenes (2020, methodology): “A few unsupervised models such as Pix2Shape [39], GQN [11], and neural rendering methods [47] have been proposed.”
BARF: Bundle-Adjusting Neural Radiance Fields (2021, methodology): “…recent advances on photorealistic view synthesis have opted to recover an intermediate dense 3D-aware representation ( e.g . depth [15, 61], multi-plane images [71, 51, 55], or volume density [27, 31]), followed by neural rendering techniques [14, 29, 47, 54] to synthesize the target images.”
NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections (2020, methodology): “Neural Rendering: More recently, neural rendering techniques [34] have been applied to scene reconstruction.”
PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting (2021, background): “The success of neural rendering [46, 29, 54, 31, 44, 43, 47] has generated signiﬁcant excitement.”
Neural Sparse Voxel Fields (2020, methodology): “We refer the reader to recent surveys for neural rendering (Tewari et al., 2020; Kato et al., 2020).”
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, background): “Despite great progress [44, 69, 74, 81, 83] brought by neural networks in computer vision or graphics fields for extracting 3D information from images, generating novel view images with multiview consistency from a single-view image of an object is still a challenging problem due to the limited 3D…”
Mixture of volumetric primitives for efficient neural rendering (2021, methodology): “For an extensive discussion of neural rendering applications, we refer to Tewari et al. [2020].”
Neural Scene Graphs for Dynamic Scenes (2020, background): “Implicit Scene Representations and Neural Rendering A rapidly growing body of work has explored neural rendering methods [38] for static scenes.”