Mega-NeRF: Scalable Construction of Large-Scale NeRFs for Virtual Fly- Throughs

By Haithem Turki, Deva Ramanan, Mahadev Satyanarayanan. Computer Vision and Pattern Recognition, 2021

592 citations recorded by the source corpus.

We use neural radiance fields (NeRFs) to build interac-tive 3D environments from large-scale visual captures spanning buildings or even multiple city blocks collected pri-marily from drones. In contrast to single object scenes (on which NeRFs are traditionally evaluated), our scale poses multiple challenges including (1) the need to model thou-sands of images with varying lighting conditions, each of which capture only a small subset of the scene, (2) pro-hibitively large model capacities that make it infeasible to train on a single GPU, and (3) significant challenges for fast rendering that would enable interactive fly-throughs. To address these challenges, we begin by analyzing visi-bility statistics for large-scale scenes, motivating a sparse network structure where parameters are specialized to dif-ferent regions of the scene. We introduce a simple geomet-ric clustering algorithm for data parallelism that partitions training images (or rather pixels) into different NeRF sub-modules that can be trained in parallel. We evaluate our approach on existing datasets (Quad 6k and UrbanScene3D) as well as against our own drone footage, improving training speed by 3x and PSNR by 12%. We also evaluate re-cent NeRF fast renderers on top of Mega-NeRF and intro-duce a novel method that exploits temporal coherence. Our technique achieves a 40x speedup over conventional NeRF rendering while remaining within 0.8 db in PSNR quality, exceeding the fidelity of existing fast renderers.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/80d44d92f074683ba8d4c86e1f18f0bc1a29abc4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Block-NeRF: Scalable Large Scene Neural View Synthesis (2022, methodology): “Concurrent works Mega-NeRF [65] and CityNeRF [67] utilize NeRFs to represent large scenes.”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, unknown): “Mega-NeRF [44] uses grid-based division and assigns each pixel in an image to different grids through which its ray passes.”
End-to-End Autonomous Driving: Challenges and Frontiers (2023, methodology): “These methods can produce more realistic sensor data visually than graphics-based approaches, but they have limitations such as high rendering times or requiring independent training for each scene being reconstructed [107], [115], [116], [117], [118].”
NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “Mega-NeRF [129] (December 2021) performed large scale urban reconstruction from aerial drone images.”
MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes (2023, background): “…extended to reconstruct large scenes, using crowdsourced data [Martin-Brualla et al. 2021], tiling the space with NeRF networks [Tancik et al. 2022; Turki et al. 2022], and reconstructing the scene in a warped domain where far-away regions are compressed [Barron et al. 2022; Zhang et al. 2020].”
BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (2021, background): “While this extreme multi-scale characteristic is often compounded with the “large scale” characteristic, such as delivering an entire city model containing hundreds of city blocks, in this project, we separate the large scenes in singlescale cases [35,39] from our study focus.”
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, background): “Photo-realistic and real-time rendering of 3D scenes has always been a pivotal interest in both academic research and industrial domains, with applications spanning virtual reality [51], media generation [36], and large-scale scene visualization [43, 45, 49].”
Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians (2024, methodology): “For large-scale scenes [1], [62], [63], training was set to 100k iterations, with densification ending at 50k iterations.”
DrivingGaussian: Composite Gaussian Splatting for Surrounding Dynamic Autonomous Driving Scenes (2023, unknown): “We also find that our baseline method, 3DGS [4], fails in modeling the detail areas, producing unpleasant artifacts, blurring, and unnatural colors.”
SparseNeRF: Distilling Depth Ranking for Few-shot Novel View Synthesis (2023, background): “Block-NeRF [39], CityNeRF [45], and Mega-NeRF [40] scaled the standard NeRF up to city-scale or urban-scale scenes.”