Benchmarking Denoising Algorithms with Real Photographs

By Tobias Plötz, Stefan Roth. 2017

693 citations recorded by the source corpus.

Lacking realistic ground truth data, image denoising techniques are traditionally evaluated on images corrupted by synthesized i.i.d. Gaussian noise. We aim to obviate this unrealistic setting by developing a methodology for benchmarking denoising techniques on real photographs. We capture pairs of images with different ISO values and appropriately adjusted exposure times, where the nearly noise-free low-ISO image serves as reference. To derive the ground truth, careful post-processing is needed. We correct spatial misalignment, cope with inaccuracies in the exposure parameters through a linear intensity transform based on a novel heteroscedastic Tobit regression model, and remove residual low-frequency bias that stems, e.g., from minor illumination changes. We then capture a novel benchmark dataset, the Darmstadt Noise Dataset (DND), with consumer cameras of differing sensor sizes. One interesting finding is that various recent techniques that perform well on synthetic noise are clearly outperformed by BM3D on photographs with real noise. Our benchmark delineates realistic evaluation scenarios that deviate strongly from those commonly used in the scientific literature.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Media Technology, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Media Technology

In Evaluation & Benchmarks

Citations in the corpus

Deep Image Prior (2017, methodology): “To validate if the deep image prior is suitable for denoising images corrupted with real-world non-Gaussian noise we use the benchmark of [46].”
Toward Convolutional Blind Denoising of Real Photographs (2018, methodology): “5 dB gain on PSNR can be attained by CBDNet on DND [45].”
Image Blind Denoising with Generative Adversarial Network Based Noise Modeling (2018, methodology): “However, such a paired training dataset would be unavailable or hard to derive in reality [24].”
FFDNet: Toward a Fast and Flexible Solution for CNN-Based Image Denoising (2017, background): “However, when the noise level is high, the noise will be not zero-mean any more due to clipping effects [51].”
CycleISP: Real Image Restoration via Improved Data Synthesis (2020, methodology): “Table 3: Denoising sRGB images of the DND benchmark dataset [44].”
Advancing Image Understanding in Poor Visibility Environments: A Collective Benchmark Study (2019, neutral): “Therefore, many newer and larger datasets are presented for image and video restoration, such as DIV2K [54] and MANGA109 [55] for image super-resolution, PolyU [56] and Darmstadt [57] for denoising, RawInDark [58] and LOL dataset [59] for low light enhancement, HazeRD [60], OHAZE [61] and IHAZE [62]”
Multi-Stage Progressive Image Restoration (2021, methodology): “First example is from DND [60] and the others from SIDD [1].”
Restormer: Efficient Transformer for High-Resolution Image Restoration (2021, methodology): “Real image denoising on SIDD [1] and DND [60] datasets.”
Real Image Denoising With Feature Attention (2019, methodology): “RNI15 [35], DND [46], Nam [42] and SSID [1].”
Variational Denoising Network: Toward Blind Noise Modeling and Removal (2019, methodology): “In this part, we evaluate the performance of VDN on real blind denoising task, including two banchmark datasets: DND [31] and SIDD [1].”
Uformer: A General U-Shaped Transformer for Image Restoration (2021, background): “SIDD [1] DND [46] Method PSNR ↑ SSIM ↑ PSNR ↑ SSIM ↑ BM3D [13] 25.”
Deep Learning on Image Denoising: An overview (2019, methodology): “The color-noisy image dataset included CBSD68, Kodak24 [57], McMaster [263], cc [160], DND [171], NC12 [112], SIDD [3] and Nam [160].”
NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images (2021, background): “In 2017, Pl¨otz and Roth [38] established a real raw image denoising benchmark, which showed that these deep denoisers failed to generalize beyond the synthetic data used during training and were outperformed by standard non-learned methods, such as BM3D [11].”
Spatial-Adaptive Network for Single Image Denoising (2020, positive): “Real noisy images To conduct comparisons on real noisy images, we choose DND [25], SIDD [1] and Nam [22] as test datasets.”
Image denoising using deep CNN with batch renormalization (2020, background): “It is known that real noisy images are usually captured by cameras of different types with different ISO values [46].”
Learning Enriched Features for Real Image Restoration and Enhancement (2020, methodology): “(1) DND [79] consists of 50 images captured with four consumer cameras.”
Generating Training Data for Denoising Real RGB Images via Camera Pipeline Simulation (2019, methodology): “We provide bilinear demosaicking as well because it has been widely used in the recent camera simulation literature [5, 25, 13].”
A High-Quality Denoising Dataset for Smartphone Cameras (2018, methodology): “However, most of the DND images have relatively low levels of noise and normal lighting conditions.”
Learning Enriched Features for Fast Image Restoration and Enhancement (2022, methodology): “(1) DND [115] consists of 50 images captured with four consumer cameras.”