Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering

By Vincent Sitzmann, Semon Rezchikov, William T. Freeman, Josh Tenenbaum, Frédo Durand. Neural Information Processing Systems, 2021

404 citations recorded by the source corpus.

Inferring representations of 3D scenes from 2D observations is a fundamental problem of computer graphics, computer vision, and artificial intelligence. Emerging 3D-structured neural scene representations are a promising approach to 3D scene understanding. In this work, we propose a novel neural scene representation, Light Field Networks or LFNs, which represent both geometry and appearance of the underlying 3D scene in a 360-degree, four-dimensional light field parameterized via a neural implicit representation. Rendering a ray from an LFN requires only a single network evaluation, as opposed to hundreds of evaluations per ray for ray-marching or volumetric based renderers in 3D-structured neural scene representations. In the setting of simple scenes, we leverage meta-learning to learn a prior over LFNs that enables multi-view consistent light field reconstruction from as little as a single image observation. This results in dramatic reductions in time and memory complexity, and enables real-time rendering. The cost of storing a 360-degree light field via an LFN is two orders of magnitude lower than conventional methods such as the Lumigraph. Utilizing the analytical differentiability of neural implicit representations and a novel parameterization of light space, we further demonstrate the extraction of sparse depth maps from LFNs.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/fb0cd71b23b19a452b6981de3b972ac891a9f60d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, neutral): “The original authors offered ﬁne-tuned and from-scratch conﬁgurations, but in practice, the pretrained networks are used as is.”
Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis (2023, background): “One group of methods focuses on improving the sampling strategy to reduce the number of point queries [4, 68, 73] or applies light field-based formulation [3, 25, 53, 54, 79, 81, 89].”
GRM: Large Gaussian Reconstruction Model for Efficient 3D Reconstruction and Generation (2024, methodology): “For a given input image I v ∈ R H × W × 3 , we first inject the camera information to every pixel following [85,106] with Plücker embedding [34, 85].”
Fast Dynamic Radiance Fields with Time-Aware Neural Voxels (2022, methodology): “Some works propose to reduce inference times for acceleration by improving sampling strategies [Arandjelović and Zisserman 2021; Fang et al. 2021; Lindell et al. 2021; Neff et al. 2021; Piala and Clark 2021] or introducing efficient rendering techniques [Sitzmann et al. 2021].”
Kubric: A scalable dataset generator (2022, methodology): “We benchmark two recently proposed models on this dataset: Light Field Networks [86], which parameterizes”
LightGaussian: Unbounded 3D Gaussian Compression with 15x Reduction and 200+ FPS (2023, methodology): “Follow-ups either explore ray re-parameterizations[4, 5], explicit spatial data structures [10, 21, 31, 39, 45, 55, 66], caching and distillation [22, 28, 48, 60], or ray-based representations [2, 54] for speeding up.”
ViewCrafter: Taming Video Diffusion Models for High-fidelity Novel View Synthesis (2024, background): “Recently, CamCo [57] and CameraCtrl [58] introduced Pl¨ucker coordinates [59] in video diffusion models for camera motion control.”
MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes (2023, methodology): “Light field coordinates circumvent this problem and require just one MLP evaluation per ray [Attal et al. 2022; Cao et al. 2023; Li et al. 2022c; Sitzmann et al. 2021; Wang et al. 2022b].”
Advances in Neural Rendering (2021, background): “Light Field Networks [SRF ∗ 21] takes this a step further, optimizing an MLP to directly encode the mapping from an input ray to an output color (the scene’s light ﬁeld).”
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation (2021, background): “Most saliently, such ﬁelds have been proposed to represent 3D geometry [5, 25, 27, 29, 33], appearance [22, 24, 34, 37, 38, 44, 47], and tactile properties [11].”
CameraCtrl: Enabling Camera Control for Text-to-Video Generation (2024, positive): “Concretely, we adopt Plücker embeddings (Sitzmann et al., 2021) as the primary form of camera pose conditioning.”
Depth Anything 3: Recovering the Visual Space from Any Views (2025, neutral): “Feed-Forward Novel View Synthesis Novel view synthesis (NVS) has long been a core problem in computer vision and graphics [8, 34, 49], and interest has increased with the rise of neural rendering [28, 44, 57, 77, 78].”
MVSplat: Efficient 3D Gaussian Splatting from Sparse Multi-View Images (2024, background): “…neural scene representations, e.g ., Scene Representation Networks (SRN) [34], Neural Radiance Fields (NeRF) [24] and Light Filed Networks (LFN) [33], these methods are still not satisfactory for practical applications due to expensive per-scene optimization [26,39,43], high memory cost…”
Scene Representation Transformer: Geometry-Free Novel View Synthesis Through Set-Latent Scene Representations (2021, methodology): “Note that crucially, the set-latent scene representation scales in size with the amount of input information rather than being ﬁxed-size [16, 20, 38].”
Novel View Synthesis with Diffusion Models (2022, methodology): “In recent literature, geometry-free approaches (i.e., methods without explicit geometric inductive biases like those introduced by volume rendering) such as Light Field Networks (LFN) (Sitzmann et al., 2021) and Scene Representation Transformers (SRT) (Sajjadi et al., 2021) have achieved re-sults co”
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, background): “Despite several works have been proposed to accelerate or mitigate the intensive volumetric ray-marching, e.g . using proposal network [4], baking technique [11, 19], and surface rendering [41].”
Continuous 3D Perception Model with Persistent State (2025, neutral): “Regression-based few-shot novel view synthesis approaches [39, 68, 69, 96] typically only generalize across a class of 3D scenes.”
PixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction (2023, background): “While the above methods infer inter-pretable 3D representations in the form of signed distances or radiance fields, recent light field scene representations trade interpretability for faster rendering [10, 37, 43, 46, 47].”