Advances in Neural Rendering

By A. Tewari, O. Fried, Justus Thies, Vincent Sitzmann, S. Lombardi, Z. Xu, T. Simon, Matthias Nießner, Edgar Tretschk, L. Liu and 12 more. SIGGRAPH Courses, 2021

551 citations recorded by the source corpus.

Synthesizing photo‐realistic images and videos is at the heart of computer graphics and has been the focus of decades of research. Traditionally, synthetic images of a scene are generated using rendering algorithms such as rasterization or ray tracing, which take specifically defined representations of geometry and material properties as input. Collectively, these inputs define the actual scene and what is rendered, and are referred to as the scene representation (where a scene consists of one or more objects). Example scene representations are triangle meshes with accompanied textures (e.g., created by an artist), point clouds (e.g., from a depth sensor), volumetric grids (e.g., from a CT scan), or implicit surface functions (e.g., truncated signed distance fields). The reconstruction of such a scene representation from observations using differentiable rendering losses is known as inverse graphics or inverse rendering. Neural rendering is closely related, and combines ideas from classical computer graphics and machine learning to create algorithms for synthesizing images from real‐world observations. Neural rendering is a leap forward towards the goal of synthesizing photo‐realistic image and video content. In recent years, we have seen immense progress in this field through hundreds of publications that show different ways to inject learnable components into the rendering pipeline. This state‐of‐the‐art report on advances in neural rendering focuses on methods that combine classical rendering principles with learned 3D scene representations, often now referred to as neural scene representations. A key advantage of these methods is that they are 3D‐consistent by design, enabling applications such as novel viewpoint synthesis of a captured scene. In addition to methods that handle static scenes, we cover neural scene representations for modeling non‐rigidly deforming objects and scene editing and composition. While most of these approaches are scene‐specific, we also discuss techniques that generalize across object classes and can be used for generative tasks. In addition to reviewing these state‐of‐the‐art methods, we provide an overview of fundamental concepts and definitions used in the current literature. We conclude with a discussion on open challenges and social implications.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0b5b6598e3e108147842f35ff66a95d989f9ec89

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

LION: Latent Point Diffusion Models for 3D Shape Generation (2022, background): “A promising extension would be to include image-based training by incorporating neural or differentiable rendering [17, 107–111] and to also synthesize textures [16, 112–114].”
Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations (2021, unknown): “1 and [40]).”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, background): “NeRF [33] have made significant advancements [55, 60].”
NeRF: Representing scenes as neural radiance ﬁelds for view synthesis (unknown): “A number of surveys on NeRF have been published [ 13, 70 ] which we refer readers to, and even the popular press has highlighted the method [ 27 ].”
Object Scene Representation Transformer (2022, methodology): “We present Object Scene Representation Transformer (OSRT) , an efﬁcient and scalable architecture for unsupervised neural scene decomposition and rendering.”
BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (2021, background): “Neural volumetric representations [21,24,16,22,15,37] have demonstrated remarkable capability in representing 3D objects and scenes from images.”
3D Neural Field Generation Using Triplane Diffusion (2022, unknown): “Implicit neural representations, or neural fields, hold the SOTA for 3D scene representation [74, 80].”
3D Gaussian Splatting for Real-Time Radiance Field Rendering (2023, methodology): “Recent neural rendering algorithms [Tewari et al. 2022] vastly reduce such artifacts and avoid the overwhelming cost of storing all input images on the GPU, outperforming these methods on most fronts.”
NeRF-Editing: Geometry Editing of Neural Radiance Fields (2022, background): “Recently, used both as a component in deep neural network pipelines and as a standalone rendering pipeline, neural rendering has achieved immense progress, which is comprehensively summarized in [64, 65].”
PixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction (2023, background): “Advancements in neural rendering [50] and neural fields [29, 42, 57] have rev-olutionized 3D reconstruction and novel view synthesis from collections of posed images.”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “[19] published a state-of-theart report on advances in Neural Rendering with a focus on NeRF models.”
StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, background): “We refer readers to [70] for more comprehensive review.”
Neural 3D Video Synthesis from Multi-view Video (2021, unknown): “[56] Ayush Tewari, Ohad Fried, Justus Thies, Vincent Sitzmann, Stephen Lombardi, Zexiang Xu, Tomas Simon, Matthias Nießner, Edgar Tretschk, L.”
Generative Novel View Synthesis with 3D-Aware Diffusion Models (2023, methodology): “[78] Ayush Tewari, Justus Thies, Ben Mildenhall, Pratul Srinivasan, Edgar Tretschk, W Yifan, Christoph Lassner, Vincent Sitzmann, Ricardo Martin-Brualla, Stephen Lombardi, et al.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, background): “For a review of this emerging space we point the reader to the reports by Kato et al. [34], Tewari et al. [80], and Xie et al. [89].”