NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images

By Ben Mildenhall, Peter Hedman, Ricardo Martin-Brualla, Pratul P. Srinivasan, Jonathan T. Barron. Computer Vision and Pattern Recognition, 2021

491 citations recorded by the source corpus.

Neural Radiance Fields (NeRF) is a technique for high quality novel view synthesis from a collection of posed input images. Like most view synthesis methods, NeRF uses tonemapped low dynamic range (LDR) as input; these images have been processed by a lossy camera pipeline that smooths detail, clips highlights, and distorts the simple noise distribution of raw sensor data. We modify NeRF to instead train directly on linear raw images, preserving the scene's full dynamic range. By rendering raw output images from the resulting NeRF, we can perform novel high dynamic range (HDR) view synthesis tasks. In addition to changing the camera viewpoint, we can manipulate focus, exposure, and tonemapping after the fact. Although a single raw image appears significantly more noisy than a postprocessed one, we show that NeRF is highly robust to the zeromean distribution of raw noise. When optimized over many noisy raw inputs (25–200), NeRF produces a scene representation so accurate that its rendered novel views outperform dedicated single and multi-image deep raw denoisers run on the same wide baseline input images. As a result, our method, which we call RawNeRF, can reconstruct scenes from extremely noisy images captured in near-darkness.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/de744193c2c21f5e518b71a804892498a9b76925

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

NeRF: Neural Radiance Field in 3D Vision, A Comprehensive Review (2022, unknown): “Denoising/Deblurring RawNeRF [38], DeblurNeRF [134], NaN [135]”
HexPlane: A Fast Representation for Dynamic Scenes (2023, background): “NeRF [46] and its variants [2, 3, 44, 48, 74, 76, 85, 92] show impressive results on novel view synthesis [9,80,87,99] and many other applications including 3D reconstruction [42, 72, 90, 94, 100], semantic segmentation [27,59,98], generative model [5,6,10, 49,62,82], and 3D content creation…”
Compact 3D Gaussian Representation for Radiance Field (2023, background): “…inception, various works have been proposed to enhance performance in diverse scenarios, such as different resolutions of reconstruction [2, 3], the reduced number of training samples [32, 36, 42, 44, 47], and reconstruction of large realistic scenes [29, 41] and dynamic scenes [12, 23, 24, 33].”
Mip-Splatting: Alias-Free 3D Gaussian Splatting (2023, methodology): “Our 2D Mip filter mimics the 2D box filter inherent to the actual physical imaging process [29, 37, 48], by approximating it with a 2D Gaussian low pass filter.”
FreeNeRF: Improving Few-Shot Neural Rendering with Free Frequency Regularization (2023, background): “The seminal work, Neural Radiance Fields (NeRF) [21], has been widely studied and advanced in a variety of applications [2, 3, 13, 19, 23, 25, 32], including novel view synthesis [18, 21], 3D generation [10, 25], deformation [23,26,28], video [7,14,15,24,35].”
Instruct-NeRF2NeRF: Editing 3D Scenes with Instructions (2023, background): “One approach is to impose physics-based inductive biases in its optimization process to enable changes in materials or scene lighting [44, 1, 40, 27, 25].”
Zip-NeRF: Anti-Aliased Grid-Based Neural Radiance Fields (2023, methodology): “NeRF has proven to be an effective tool for tasks such as view synthesis [19], generative media [27], robotics [35], and computational photography [20].”