PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting

By Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, Noah Snavely. Computer Vision and Pattern Recognition, 2021

454 citations recorded by the source corpus.

We present PhySG, an end-to-end inverse rendering pipeline that includes a fully differentiable renderer, and can reconstruct geometry, materials, and illumination from scratch from a set of images. Our framework represents specular BRDFs and environmental illumination using mixtures of spherical Gaussians, and represents geometry as a signed distance function parameterized as a Multi-Layer Perceptron. The use of spherical Gaussians allows us to efficiently solve for approximate light transport, and our method works on scenes with challenging non-Lambertian reflectance captured under natural, static illumination. We demonstrate, with both synthetic and real data, that our reconstructions not only enable rendering of novel viewpoints, but also physics-based appearance editing of materials and illumination.

View Paper: https://www.semanticscholar.org/paper/50bca2632e973fc6a60cdd7da1d64f3fc5ddd9b1

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

NeRD: Neural Reflectance Decomposition from Image Collections (2020, background): “Some concurrent works [50, 61] also try to estimate BRDFs in neural volumes.”
SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion (2024, methodology): “We base our rendering model on Spherical Gaussians (SG) [4, 60].”
NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction (2021, background): “Recent works represent surfaces as signed distance functions (SDF) [46, 49, 17, 22] or occupancy [29, 30].”
SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering (2023, methodology): “Retrieving such an editable mesh for realistic rendering is done within minutes with our method, compared to hours with the state-of-the-art method on SDFs, while providing a better rendering quality.”
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (2022, methodology): “Inspired by [29, 33, 9], we instead employ a spherical Gaussian (SG) rendering framework [9], which approximates every term in Eq.”
Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields (2021, unknown): “For comparisons to PhySG [45], we use the ofﬁ-cial PyTorch code open-sourced by the authors.”