NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections

By Ricardo Martin-Brualla, Noha Radwan, Mehdi S. M. Sajjadi, Jonathan T. Barron, Alexey Dosovitskiy, Daniel Duckworth. 2020

1,885 citations recorded by the source corpus.

We present a learning-based method for synthesizing novel views of complex scenes using only unstructured collections of in-the-wild photographs. We build on Neural Radiance Fields (NeRF), which uses the weights of a multi-layer perceptron to model the density and color of a scene as a function of 3D coordinates. While NeRF works well on images of static subjects captured under controlled settings, it is incapable of modeling many ubiquitous, real-world phenomena in uncontrolled images, such as variable illumination or transient occluders. We introduce a series of extensions to NeRF to address these issues, thereby enabling accurate reconstructions from unstructured image collections taken from the internet. We apply our system, dubbed NeRF-W, to internet photo collections of famous landmarks, and demonstrate temporally consistent novel view renderings that are significantly closer to photorealism than the prior state of the art.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

GAN Inversion: A Survey (2021, methodology): “It has been used for volumetric performance capture [169], [170], [171], novel-view synthesis [172], [172], face shape generation [173], object modeling [174], and human reconstruction [175], [176], [177], [178].”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “However,it is significantly harder to obtain 3D object datasets, as it requires special techniques such as photogrammetry [113] and other neural rendering approaches [114] to be synthesized before it can be annotated.”
Dynamic View Synthesis from Dynamic Monocular Video (2021, methodology): “The method in [32] extends NeRF for handling crowdsourced photos that contain lighting variations and transient objects.”
Learned Initializations for Optimizing Coordinate-Based Neural Representations (2020, background): “NeRF in the Wild [21] explores these shortTask CelebA Imagenette Text SDF”
RegNeRF: Regularizing Neural Radiance Fields for View Synthesis from Sparse Inputs (2021, background): “…representations [7, 33, 34, 44] have become a popular representation for various tasks such as 3D reconstruction [1,7,13,14,33,39,42,44,45,48,51,55,57], 3D-aware generative modelling [5, 9, 15, 16, 32, 37, 38, 41, 49, 64], and novel-view synthesis [2, 3, 12, 22, 24, 27, 31, 36, 40, 43, 52, 60, 61].”
Octree-GS: Towards Consistent Real-time Rendering with LOD-Structured 3D Gaussians (2024, methodology): “To address this, and following the approach of prior NeRF papers [57], [58], we integrate Generative Latent Optimization (GLO) [59] to generate the color of Gaussian primitives.”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, result): “For example, this could be addressed by a latent code factoring out such variations, similarly done in NeRF-W [44].”
What's Behind the Couch? Directed Ray Distance Functions (DRDF) for 3D Scene Reconstruction (2021, methodology): “These implicit functions have also been used in view synthesis [34, 36, 56, 57], which differs from our work in goals.”
VastGaussian: Vast 3D Gaussians for Large Scene Reconstruction (2024, positive): “To address this problem, some NeRF-based methods [29, 41, 44, 61] concatenate an appearance embedding to point-based features in pixel-wise ray-marching, and feed them into the radiance MLP to obtain the final color.”
AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesis (2021, methodology): “Follow-up works extend this idea by using in-the-wild training data including appearance interpolation [29], introducing deformable neural radiance ﬁelds to represent non-rigidly moving objects [31, 36], and optimizing NeRF without pre-computed camera parameters [52].”
Pixel-aligned Volumetric Avatars (2021, background): “NeRF-W [14] extends the work of [16] to a conditional setting to models scenes under different lighting with same underlying geometry.”
GIRAFFE: Representing Scenes as Compositional Generative Neural Feature Fields (2020, methodology): “3D Volume Rendering: While previous works [47,57,61, 77] volume render an RGB color value, we extend this formulation to rendering an M f -dimensional feature vector f .”
4D Gaussian Splatting for Real-Time Dynamic Scene Rendering (2023, background): “Meanwhile, differential splatting [42] creates a relationship among 3D Gaussians on the rendered pixel directly, therefore, each rendered pixel may influenced by different 3D Gaussians every timestamp.”
Ref-NeRF: Structured View-Dependent Appearance for Neural Radiance Fields (2021, positive): “NeRF has inspired many subsequent works, which extend its neural volumetric scene representation to application domains including dynamic and deformable scenes [24], avatar animation [10, 25], and even phototourism [19].”
Direct Voxel Grid Optimization: Super-fast Convergence for Radiance Fields Reconstruction (2021, background): “, relighting [2, 3, 50, 70], deformation [13, 38–40, 55], self-calibration [19, 27, 28, 35, 61], meta-learning [52], dynamic scene modeling [14, 25, 33, 41, 64], and generative modeling [5, 22, 44].”