Unprocessing Images for Learned Raw Denoising

By Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, Jonathan T. Barron. 2018

540 citations recorded by the source corpus.

Machine learning techniques work best when the data used for training resembles the data used for evaluation. This holds true for learned single-image denoising algorithms, which are applied to real raw camera sensor readings but, due to practical constraints, are often trained on synthetic image data. Though it is understood that generalizing from synthetic to real images requires careful consideration of the noise properties of camera sensors, the other aspects of an image processing pipeline (such as gain, color correction, and tone mapping) are often overlooked, despite their significant effect on how raw measurements are transformed into finished images. To address this, we present a technique to “unprocess” images by inverting each step of an image processing pipeline, thereby allowing us to synthesize realistic raw sensor measurements from commonly available Internet photos. We additionally model the relevant components of an image processing pipeline when evaluating our loss function, which allows training to be aware of all relevant photometric processing that will occur after denoising. By unprocessing and processing training data and model outputs in this way, we are able to train a simple convolutional neural network that has 14%-38% lower error rates and is 9×-18× faster than the previous state of the art on the Darmstadt Noise Dataset, and generalizes to sensors outside of that dataset as well.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Designing a Practical Degradation Model for Deep Blind Image Super-Resolution (2021, methodology): “The raw image noise model is borrowed from [6].”
Multi-Stage Progressive Image Restoration (2021, methodology): “Numerous network modules and functional units for image restoration have been developed including recursive residual learning [4, 95], dilated convolutions [4, 81], attention mechanisms [17, 86, 96], dense connections [73, 75, 97], encoder-decoders [7, 13, 43, 65], and generative models [44, 62, 90,”
Real Image Denoising With Feature Attention (2019, background): “Recently, many algorithms focused on blind denoising on real-noisy images [50, 31, 12].”
Learning to Restore Low-Light Images via Decomposition-and-Enhancement (2020, methodology): “However, the linear camera raw data is significantly different from the non-linear sRGB data, particularly in terms of noise [2] and image intensity [46].”
Learning Enriched Features for Real Image Restoration and Enhancement (2020, result): “Recently, deep learningbased approaches [11,5,9,35,39,80,120,121,119] make significant advances in image denoising, yielding favorable results than those of the hand-crafted methods.”
Seeing Motion in the Dark (2019, methodology): “validated that a carefully designed raw image processing system can significantly improve results without significant changes in network structure and loss function [7].”
Deep Generalized Unfolding Networks for Image Restoration (2022, methodology): “In addition to estimating the clean image with a fixed scale, some hourglassshaped networks [6,24,33,75,79] were proposed to explore multi-scale feature maps for IR.”
Generating Training Data for Denoising Real RGB Images via Camera Pipeline Simulation (2019, methodology): “We provide bilinear demosaicking as well because it has been widely used in the recent camera simulation literature [5, 25, 13].”
Benchmarking Low-Light Image Enhancement and Beyond (2021, methodology): “Differently, we additionally consider noise modeling at the RAW image level following (Brooks et al. 2019).”
NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections (2020, background): “This issue is exacerbated by photographic imaging pipelines, as variation in auto-exposure settings, white balance, and tone-mapping across photographs may result in additional photometric inconsistencies [4].”
Artificial intelligence in the creative industries: a review (2020, background): “[133] have developed specific neural networks to denoise real images through ‘unprocessing’, i.”
CycleISP: Real Image Restoration via Improved Data Synthesis (2020, methodology): “For a fair comparison, we use the same network architecture (U-Net) and noise model as of [7].”
NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images (2021, methodology): “Subsequent work on both single [5, 9] and multi-image [8, 19, 38, 53] denoising demonstrated the benefits of training networks to operate directly on noisy raw input data.”
FastDVDnet: Towards Real-Time Deep Video Denoising Without Flow Estimation (2019, background): “The use of a noise map has been shown to improve denoising performance, particularly when treating spatially variant noise [4].”
Deep Learning on Image Denoising: An overview (2019, unknown): “] CNN Real noisy image denoising CNN for real noisy image denoising Green et al. (2018) [67] CNN CT image denoising, real noisy image denoising CNN for real noisy image denoising Brooks et al. (2019) [24] CNN Real noisy image denoising CNN with image processing pipeline for real noisy image denoisin”
Variational Denoising Network: Toward Blind Noise Modeling and Removal (2019, background): “distribution for the noise as conventional [27, 13, 16, 38], which largely deviates the spatial variant and signal-depend characteristics of the real noise [41, 8], we models the noise as a non-i.”