MobileNeRF: Exploiting the Polygon Rasterization Pipeline for Efficient Neural Field Rendering on Mobile Architectures

By Zhiqin Chen, Thomas A. Funkhouser, Peter Hedman, Andrea Tagliasacchi. Computer Vision and Pattern Recognition, 2022

454 citations recorded by the source corpus.

Neural Radiance Fields (NeRFs) have demonstrated amazing ability to synthesize images of 3D scenes from novel views. However, they rely upon specialized volumetric rendering algorithms based on ray marching that are mismatched to the capabilities of widely deployed graphics hardware. This paper introduces a new NeRF representation based on textured polygons that can synthesize novel images efficiently with standard rendering pipelines. The NeRF is represented as a set of polygons with textures representing binary opacities and feature vectors. Traditional rendering of the polygons with a z-buffer yields an image with features at every pixel, which are interpreted by a small, view-dependent MLP running in a fragment shader to produce a final pixel color. This approach enables NeRFs to be rendered with the traditional polygon rasterization pipeline, which provides massive pixel-level parallelism, achieving interactive frame rates on a wide range of compute platforms, including mobile phones. Project page: https://mobile-nerf.github.io

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/b31c2967451722780e92604532e19cdd3f0f49fb

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

GaussianEditor: Swift and Controllable 3D Editing with Gaussian Splatting (2023, unknown): “However, optimizing NeRF can be time-consuming, despite its wide usage in 3D reconstruction [3, 6, 13, 21, 44] and generation [22, 35] tasks.”
2D Gaussian Splatting for Geometrically Accurate Radiance Fields (2024, unknown): “…3DGS has been quickly extended with respect to multiple domains, including anti-aliasing rendering [Yu et al. 2024], material modeling [Jiang et al. 2023; Shi et al. 2023], dynamic scene reconstruction [Yan et al. 2023], and animatable avatar creation [Qian et al. 2023; Zielonka et al. 2023].”
MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes (2023, methodology): “…view synthesis methods (NeRF [Mildenhall et al. 2020], mip-NeRF 360 [Barron et al. 2022], Stable View Synthesis [Riegler and Koltun 2021], and Instant-NGP [Müller et al. 2022]), and real-time ones (Deep Blend-ing [Hedman et al. 2018], Mobile-NeRF [Chen et al. 2023], and SNeRG [Hedman et al. 2021]).”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, unknown): “…becomes a popular 3D representation for photo-realistic novel-view synthesis in recent years [35], many variants are proposed to improve quality [2–4, 24, 45, 47–49, 57], increase speed [8, 9, 11, 14, 20, 32, 36, 37, 40, 43, 46, 58, 60], extend to dynamic scenes [7, 15, 18, 25, 27, 50], and so on.”
Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction (2023, methodology): “A prominent technique [8] in this category transforms NeRF scenes into an amalgamation of coarse meshes and feature textures, thereby enhancing rendering velocity in contemporary mobile graphics pipelines.”
DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation (2023, methodology): “Although NeRF has become widely used in both 3D reconstruction (Barron et al., 2022; Li et al., 2023d; Chen et al., 2022; Hedman et al., 2021) and generation (Poole et al., 2022; Lin et al., 2023; Chan et al., 2022), optimizing NeRF can be time-consuming.”
SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering (2023, methodology): “More details about this parameterisation are given in the supplementary material.”
UniSim: A Neural Closed-Loop Sensor Simulator (2023, methodology): “We believe the run-time can be further optimized with recent advancements [11,97] and code optimization ( e.g ., customized CUDA kernels).”
Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering (2023, background): “Despite several works have been proposed to accelerate or mitigate the intensive volumetric ray-marching, e.g . using proposal network [4], baking technique [11, 19], and surface rendering [41].”
BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis (2023, methodology): “Perhaps surprisingly, replacing our SG appearance model with the small view-dependent MLP used by both SNeRG [Hed-man et al. 2021] and MobileNeRF [Chen et al. 2022a] significantly reduces rendering quality and yields error metrics that are roughly comparable to the “1 Spherical Gaussian” ablation.”