BakedSDF: Meshing Neural SDFs for Real-Time View Synthesis

By Lior Yariv, Peter Hedman, Christian Reiser, Dor Verbin, Pratul P. Srinivasan, Richard Szeliski, Jonathan T. Barron, Ben Mildenhall. International Conference on Computer Graphics and Interactive Techniques, 2023

311 citations recorded by the source corpus.

We present a method for reconstructing high-quality meshes of large unbounded real-world scenes suitable for photorealistic novel view synthesis. We first optimize a hybrid neural volume-surface scene representation designed to have well-behaved level sets that correspond to surfaces in the scene. We then bake this representation into a high-quality triangle mesh, which we equip with a simple and fast view-dependent appearance model based on spherical Gaussians. Finally, we optimize this baked representation to best reproduce the captured viewpoints, resulting in a model that can leverage accelerated polygon rasterization pipelines for real-time view synthesis on commodity hardware. Our approach outperforms previous scene representations for real-time rendering in terms of accuracy, speed, and power consumption, and produces high quality meshes that enable applications such as appearance editing and physical simulation.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/bd8d5fe2c7b35241410262a0a26edfdca9673dc6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

MERF: Memory-Efficient Radiance Fields for Real-time View Synthesis in Unbounded Scenes (2023, methodology): “2022a], extract a mesh after training [Kellnhofer et al. 2021; Yariv et al. 2023] and/or restrict NeRF evaluation to planes or polygons [Chen et al.”
SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering (2023, methodology): “In particular, the recent BakedSDF [39] re-constructs high quality meshes by optimizing a full neural SDF model, baking it into a high-resolution triangle mesh that combines mesh rendering for interpolating features and deep learning to translate these features into images, and ﬁnally optimizes a…”
Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields (2023, methodology): “We evaluate NeRF [21], mip-NeRF [3], NeRF++ [38], mipNeRF 360, iNGP [23] (using carefully tuned hyperparameters taken from recent work [34]), and a baseline in which we naively combine mip-NeRF 360 and iNGP without any of this paper’s contributions.”
2D Gaussian Splatting for Geometrically Accurate Radiance Fields (2024, unknown): “…3DGS has been quickly extended with respect to multiple domains, including anti-aliasing rendering [Yu et al. 2024], material modeling [Jiang et al. 2023; Shi et al. 2023], dynamic scene reconstruction [Yan et al. 2023], and animatable avatar creation [Qian et al. 2023; Zielonka et al. 2023].”
Recent advances in 3D Gaussian splatting (2024, unknown): “While significant efforts [2–4] have been made to accelerate it to facilitate its applications on common devices, it is still hard to find a robust method that can both train a NeRF quickly enough ( ⩽ 1 hour) on a consumer-level GPU and render a 3D scene at an interactive frame rate ( ∼ 30 FPS) on…”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, unknown): “…becomes a popular 3D representation for photo-realistic novel-view synthesis in recent years [35], many variants are proposed to improve quality [2–4, 24, 45, 47–49, 57], increase speed [8, 9, 11, 14, 20, 32, 36, 37, 40, 43, 46, 58, 60], extend to dynamic scenes [7, 15, 18, 25, 27, 50], and so on.”
Mip-Splatting: Alias-Free 3D Gaussian Splatting (2023, methodology): “Subsequent methods [16, 40, 41, 52, 54] distill a pretrained NeRF into a sparse representation, enabling real-time rendering of NeRFs.”