Local light field fusion

By Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, Abhishek Kar. ACM Transactions on Graphics, 2019

1,308 citations recorded by the source corpus.

We present a practical and robust deep learning solution for capturing and rendering novel views of complex real world scenes for virtual exploration. Previous approaches either require intractably dense view sampling or provide little to no guidance for how users should sample views of a scene to reliably render high-quality novel views. Instead, we propose an algorithm for view synthesis from an irregular grid of sampled views that first expands each sampled view into a local light field via a multiplane image (MPI) scene representation, then renders novel views by blending adjacent local light fields. We extend traditional plenoptic sampling theory to derive a bound that specifies precisely how densely users should sample views of a given scene when using our algorithm. In practice, we apply this bound to capture and render views of real world scenes that achieve the perceptual quality of Nyquist rate view sampling while using up to 4000X fewer views. We demonstrate our approach's practicality with an augmented reality smart-phone app that guides users to capture input images of a scene and viewers that enable realtime virtual exploration on desktop and mobile platforms.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/af39e137818ef8304ccea2ada546700de1dd2f8c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Mixture of volumetric primitives for efficient neural rendering (2021, methodology): “) [Mildenhall et al. 2019].”
D-NeRF: Neural Radiance Fields for Dynamic Scenes (2020, neutral): “More recently, deep learning based techniques [36, 16, 10, 9, 25] are able to learn a neural volumetric representation from a set of sparse images.”
Light Field Networks: Neural Scene Representations with Single-Evaluation Rendering (2021, methodology): “This has enabled a number of applications, such as novel view synthesis [7, 3, 6], 3D reconstruction [5, 3] semantic segmentation [8, 9], and SLAM [10].”
Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly- Throughs (2021, neutral): “Resolution # Images # Pixels/Rays Synthetic NeRF [22] 400 x 400 400 256,000,000 LLFF [21] 4032 x 3024 41 496,419,840 Light Field [37] 1280 x 720 214 195,910,200 Tanks and Temples [15] 1920 x 1080 283 587,658,240 Phototourism [13] 919 x 794 1708 1,149,113,846”
Nerfstudio: A Modular Framework for Neural Radiance Field Development (2023, neutral): “Our goal is to provide researchers with more 360 real-world captures that are not limited to forward-facing scenes [Mildenhall et al. 2019].”
KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs (2021, neutral): “In contrast, approaches using multi-plane images (MPIs) [11,28,55,56,64,78] represent the scene as a stack of images or feature maps.”
Urban Radiance Fields (2021, background): “56] or small regions of real scenes [8, 10, 42, 43, 64, 72].”
NeRF-: Neural Radiance Fields Without Known Camera Parameters (2021, methodology): “As an alternative, volume-based representations have been proposed to directly model the appearance of the 3D space [Flynn et al. 2016; Mildenhall et al. 2019, 2020; Penner and Zhang 2017; Seitz and Dyer 1999; Sitzmann et al. 2019; Zhou et al. 2018].”
PlenOctrees for Real-time Rendering of Neural Radiance Fields (2021, background): “Volume-based representations such as voxel grids [42, 18, 24, 14, 55, 44] and multi-plane images (MPIs) [49, 35, 64, 48, 29] are a popular alternative to mesh representations due to their topology-free nature: gradient-based optimization is therefore straightforward, while rendering can still be…”
Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (2023, background): “In addition, our adaptation of the original 3DGS pipeline, which inherently generates noise-inducing floaters, poses another challenge, affecting our model’s optimal performance.”
Editing Conditional Radiance Fields (2021, background): “Prior work reasons over an explicit [11, 15, 22, 72] or discrete volumetric [20, 33, 36, 40, 44, 55, 64, 65, 67, 68, 80, 82] representation of the underlying geometry.”
CAT3D: Create Anything in 3D with Multi-View Diffusion Models (2024, methodology): “…CO3D [74] and RealEstate10K [76] are in-distribution datasets whose training splits were part of our training set (we use their test splits for evaluation), whereas DTU [78], LLFF [79] and the mip-NeRF 360 dataset [73] are out-of-distribution datasets that were not part of the training dataset.”
ScanNet++: A High-Fidelity Dataset of 3D Indoor Scenes (2023, background): “scenes Average Total scans RGB (MP) Depth Dense semantics NVS scans/scene Commodity DSLR LR HR LLFF [28] 35 12 ✗ ✗ ✗ ✗ ✓ DTU [17] 124 - 1.”
MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes (2023, background): “Similarly, multi-plane image [Flynn et al. 2019; Mildenhall et al. 2019; Wizadwongsa et al. 2021; Zhou et al. 2018] or multi-sphere image [Attal et al.”
GS-LRM: Large Reconstruction Model for 3D Gaussian Splatting (2024, background): “Early efforts adopt CNNs to estimate neural points [2,66,71] or multi-plane images (MPIs) [33,39,76], achieving rendering via point splatting or alpha compositing.”