MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes

By Christian Reiser, Richard Szeliski, Dor Verbin, Pratul P. Srinivasan, Ben Mildenhall, Andreas Geiger, Jonathan T. Barron, Peter Hedman. ACM Transactions on Graphics, 2023

297 citations recorded by the source corpus.

Neural radiance fields enable state-of-the-art photorealistic view synthesis. However, existing radiance field representations are either too compute-intensive for real-time rendering or require too much memory to scale to large scenes. We present a Memory-Efficient Radiance Field (MERF) representation that achieves real-time rendering of large-scale scenes in a browser. MERF reduces the memory consumption of prior sparse volumetric radiance fields using a combination of a sparse feature grid and high-resolution 2D feature planes. To support large-scale unbounded scenes, we introduce a novel contraction function that maps scene coordinates into a bounded volume while still allowing for efficient ray-box intersection. We design a lossless procedure for baking the parameterization used during training into a model that achieves real-time rendering while still preserving the photorealistic view synthesis quality of a volumetric radiance field.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/06ee56a1e7030fa85b3e39897a2f4ef698b2faaf

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

Mip-Splatting: Alias-Free 3D Gaussian Splatting (2023, methodology): “Subsequent methods [16, 40, 41, 52, 54] distill a pretrained NeRF into a sparse representation, enabling real-time rendering of NeRFs.”
Text-to-3D using Gaussian Splatting (2023, unknown): “After works have emerged to improve NeRF in reconstruction quality [7, 8, 76], handling large-scale [13, 40, 68, 81] and dynamic scenes [3, 47, 51, 60, 73], improving training [10, 44, 67, 79], rendering [25, 56, 80] speed and facilitating down-stream tasks [17, 46, 78, 83, 84].”
2D Gaussian Splatting for Geometrically Accurate Radiance Fields (2024, methodology): “Ad-ditionally, the rendering efficiency of NeRF has seen substantial improvements through techniques such as distillation [Reiser et al. 2021; Yu et al. 2021] and baking [Chen et al. 2023a; Hedman et al. 2021; Reiser et al. 2023; Yariv et al. 2023].”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, unknown): “…becomes a popular 3D representation for photo-realistic novel-view synthesis in recent years [35], many variants are proposed to improve quality [2–4, 24, 45, 47–49, 57], increase speed [8, 9, 11, 14, 20, 32, 36, 37, 40, 43, 46, 58, 60], extend to dynamic scenes [7, 15, 18, 25, 27, 50], and so on.”
Compressed 3D Gaussian Splatting for Accelerated Novel View Synthesis (2023, unknown): “A number of works have especially addressed memory reduction during inference, to make grid-based scene representations more suitable for low-end devices with limited video memory [ 19, 27].”