Block-NeRF: Scalable Large Scene Neural View Synthesis

By Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P. Srinivasan, Jonathan T. Barron, Henrik Kretzschmar. 2022

1,104 citations recorded by the source corpus.

We present Block-NeRF, a variant of Neural Radiance Fields that can represent large-scale environments. Specifically, we demonstrate that when scaling NeRF to render city-scale scenes spanning multiple blocks, it is vital to de-compose the scene into individually trained NeRFs. This decomposition decouples rendering time from scene size, enables rendering to scale to arbitrarily large environments, and allows per-block updates of the environment. We adopt several architectural changes to make NeRF robust to data captured over months under different environmental conditions. We add appearance embeddings, learned pose refinement, and controllable exposure to each individual NeRF, and introduce a procedure for aligning appearance between adjacent NeRFs so that they can be seamlessly combined. We build a grid of Block-NeRFs from 2.8 million images to create the largest neural scene representation to date, capable of rendering an entire neighborhood of San Francisco.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, Environmental Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In Environmental Engineering

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “…analogies to 3D vision and robotics, this can be thought of as building an on-thefly reconstruction of the sequence of events in the observable world with language, rather than other representations, such as dynamically-updated 3D meshes (Izadi et al. 2011) or neural fields (Tancik et al. 2022).”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, methodology): “These methods can produce more realistic sensor data visually than graphics-based approaches, but they have limitations such as high rendering times or requiring independent training for each scene being reconstructed [107], [115], [116], [117], [118].”
Decomposing NeRF for Editing via Feature Field Distillation (2022, background): “Regular structures such as voxelgrids or octrees [13, 48, 14, 88, 94, 89, 43, 81, 107, 59, 60] or unsupervised decomposition [73, 85, 109, 83] enable editability via manipulation of localized parameters.”
UniSim: A Neural Closed-Loop Sensor Simulator (2023, background): “Several representations [8, 42, 47, 51, 52, 61, 62, 76, 100] partition the space and model the volume more efficiently to handle large-scale unbounded outdoor scenes.”
Objaverse: A Universe of Annotated 3D Objects (2022, background): “Photogrammetry and NERF advances have enabled the commercialization of capturing high-quality 3D objects of large exteriors by taking pictures [75, 85].”
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, background): “Photo-realistic and real-time rendering of 3D scenes has always been a pivotal interest in both academic research and industrial domains, with applications spanning virtual reality [51], media generation [36], and large-scale scene visualization [43, 45, 49].”
Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (2023, background): “Block-NeRF [40] further extends NeRF to render city-scale scenes spanning multiple blocks.”
Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians (2024, methodology): “To address this, and following the approach of prior NeRF papers [57], [58], we integrate Generative Latent Optimization (GLO) [59] to generate the color of Gaussian primitives.”
Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly- Throughs (2021, background): “Concurrent to us, Urban Radiance Fields [30] (URF), CityNeRF [43], and BlockNeRF [34] target urban-scale environments.”
Compact 3D Gaussian Representation for Radiance Field (2023, background): “…inception, various works have been proposed to enhance performance in diverse scenarios, such as different resolutions of reconstruction [2, 3], the reduced number of training samples [32, 36, 42, 44, 47], and reconstruction of large realistic scenes [29, 41] and dynamic scenes [12, 23, 24, 33].”
SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis (2023, background): “range of real-world scenarios, collecting dense views of a scene is often expensive and time-consuming [39].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, neutral): “A pp lic at io ns Urban Street Level Urban Radiance Field [44], BlockNeRF [128]”
CLNeRF: Continual Learning Meets NeRF (2023, background): “Extending CLNeRF to the scale of Block-NeRF [32] is also an interesting future work.”
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (2022, background): “In the broader field of computer vision, the dataset may be used to study object detection [64]; NeRFs [80, 110, 43, 71]; segmentation, depth, and optimal flow estimation [35, 43]; generative modeling [59, 62, 61]; occlusion reasoning [34]; and pose estimation [19], among others.”
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, neutral): “We also find that our baseline method, 3DGS [4], fails in modeling the detail areas, producing unpleasant artifacts, blurring, and unnatural colors.”
Text-to-3D using Gaussian Splatting (2023, neutral): “After works have emerged to improve NeRF in reconstruction quality [7, 8, 76], handling large-scale [13, 40, 68, 81] and dynamic scenes [3, 47, 51, 60, 73], improving training [10, 44, 67, 79], rendering [25, 56, 80] speed and facilitating down-stream tasks [17, 46, 78, 83, 84].”
BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (2021, background): “While this extreme multi-scale characteristic is often compounded with the “large scale” characteristic, such as delivering an entire city model containing hundreds of city blocks, in this project, we separate the large scenes in singlescale cases [35,39] from our study focus.”
HexPlane: A Fast Representation for Dynamic Scenes (2023, background): “NeRF [46] and its variants [2, 3, 44, 48, 74, 76, 85, 92] show impressive results on novel view synthesis [9,80,87,99] and many other applications including 3D reconstruction [42, 72, 90, 94, 100], semantic segmentation [27,59,98], generative model [5,6,10, 49,62,82], and 3D content creation…”