Neural Fields in Visual Computing and Beyond

By Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, Srinath Sridhar. Computer graphics forum (Print), 2021

820 citations recorded by the source corpus.

Recent advances in machine learning have led to increased interest in solving visual computing problems using methods that employ coordinate‐based neural networks. These methods, which we call neural fields, parameterize physical properties of scenes or objects across space and time. They have seen widespread success in problems such as 3D shape and image synthesis, animation of human bodies, 3D reconstruction, and pose estimation. Rapid progress has led to numerous papers, but a consolidation of the discovered knowledge has not yet emerged. We provide context, mathematical grounding, and a review of over 250 papers in the literature on neural fields. In Part I, we focus on neural field techniques by identifying common components of neural field methods, including different conditioning, representation, forward map, architecture, and manipulation methods. In Part II, we focus on applications of neural fields to different problems in visual computing, and beyond (e.g., robotics, audio). Our review shows the breadth of topics already covered in visual computing, both historically and in current incarnations, and highlights the improved quality, flexibility, and capability brought by neural field methods. Finally, we present a companion website that acts as a living database that can be continually updated by the community.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/521026c6097b0c0b44cb97b8a3952261527bea41

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

FreeNeRF: Improving Few-Shot Neural Rendering with Free Frequency Regularization (2023, background): “Neural fields [36] use deep neural networks to represent 2D images or 3D scenes as continuous functions.”
Learning Smooth Neural Functions via Lipschitz Regularization (2022, methodology): “For an overview on neural fields, please refer to [Xie et al. 2021].”
Nerfstudio: A Modular Framework for Neural Radiance Field Development (2023, background): “…for high-quality 3D mesh exports [Oechsle et al. 2021; Wang et al. 2021a; Yariv et al. 2021], speed improvements for real-time rendering and training [Fridovich-Keil et al. 2022; Müller et al. 2022; Sun et al. 2022], 3D object generation [Poole et al. 2023], and more [Xie et al. 2022].”
Panoptic Neural Fields: A Semantic Object-Aware Neural Scene Representation (2022, neutral): “NeRF [33] have made significant advancements [55, 60].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, neutral): “In December 2021, Zhang et al. [12] published a preprint survey for Multimodal image synthesis and editing in which they dedicated a paragraph for NeRF.”
SyncDreamer: Generating Multiview-consistent Images from a Single-view Image (2023, background): “Despite great progress [44, 69, 74, 81, 83] brought by neural networks in computer vision or graphics fields for extracting 3D information from images, generating novel view images with multiview consistency from a single-view image of an object is still a challenging problem due to the limited 3D…”
Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering (2021, background): “Inspired by neural implicit representations of 3D geometry [1, 2], recent work has proposed to encode properties of 3D scenes as neural fields (also implicit- or coordinate-based representations, see [21] for an overview), neural networks that map 3D coordinates to local properties of the 3D scene a”
MonoSDF: Exploring Monocular Geometric Cues for Neural Implicit Surface Reconstruction (2022, background): “Neural implicit scene representations or neural fields [72] have recently gained popularity for representing 3D geometry due to their expressiveness and low memory footprint.”
PixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction (2023, methodology): “Advancements in neural rendering [50] and neural fields [29, 42, 57] have rev-olutionized 3D reconstruction and novel view synthesis from collections of posed images.”
A Survey on 3D Gaussian Splatting (2024, neutral): “An explicit radiance field directly represents the distribution of light in a discrete spatial structure, such as a voxel grid or a set of points [49, 212].”
3D Gaussian Splatting for Real-Time Radiance Field Rendering (2023, methodology): “For complete coverage of the field, please see the excellent recent surveys [Tewari et al. 2022; Xie et al. 2022].”
3D Neural Field Generation Using Triplane Diffusion (2022, positive): “Implicit neural representations, or neural fields, hold the SOTA for 3D scene representation [74, 80].”