Deep joint demosaicking and denoising

By Michael Gharbi, Gaurav Chaurasia, Sylvain Paris, Frédo Durand. 2016

474 citations recorded by the source corpus.

Demosaicking and denoising are the key first stages of the digital imaging pipeline but they are also a severely ill-posed problem that infers three color values per pixel from a single noisy measurement. Earlier methods rely on hand-crafted filters or priors and still exhibit disturbing visual artifacts in hard cases such as moiré or thin edges. We introduce a new data-driven approach for these challenges: we train a deep neural network on a large corpus of images instead of using hand-tuned filters. While deep learning has shown great success, its naive application using existing training datasets does not give satisfactory results for our problem because these datasets lack hard cases. To create a better training set, we present metrics to identify difficult patches and techniques for mining community photographs for such patches. Our experiments show that this network and training procedure outperform state-of-the-art both on noisy and noise-free data. Furthermore, our algorithm is an order of magnitude faster than the previous best performing techniques.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs (2017, background): “Deep visual manipulation Recently, deep neural networks have obtained promising results in various image processing tasks, such as style transfer [13], inpainting [40], colorization [56], and restoration [14].”
Real-time user-guided image colorization with learned deep priors (2017, background): “…results for various image processing tasks, such as photo enhancement (Yan et al. 2016), sketch simplification (Simo-Serra et al. 2016), style transfer (Gatys et al. 2016; Selim et al. 2016), in-painting (Pathak et al. 2016), image blending (Zhu et al. 2015) and denoising (Gharbi et al. 2016).”
3D-Aware Scene Manipulation via Inverse Graphics (2018, background): “…et al., 2016], image-to-image translation [Isola et al., 2017, Zhu et al., 2017a, Liu et al., 2017], automatic colorization [Zhang et al., 2016], inpainting [Pathak et al., 2016], attribute editing [Yan et al., 2016a], interactive editing [Zhu et al., 2016], and denoising [Gharbi et al., 2016].”
Fast Image Processing with Fully-Convolutional Networks (2017, methodology): “Deep networks have been used for denoising [39, 11, 3], super-resolution [10, 19, 40, 42, 41, 48, 50], deblurring [74], restoration of images corrupted by dirt or rain [22], example-based non-photorealistic stylization [30, 70, 40], joint image filtering [49], dehazing [64], and demosaicking [31].”
Incorporating Side Information by Adaptive Convolution (2017, methodology): “One simple solution is to add an extra image channel where each pixel holds the side information (Gharbi et al. 2016), which is equivalent to using 1st-layer ﬁlter bias terms that change with the side information.”
Seeing Motion in the Dark (2019, background): “Some networks can denoise and demosaic images jointly [21, 14] or even replace much of the image processing pipeline [9, 41].”
Underexposed Photo Enhancement Using Deep Illumination Estimation (2019, background): “Efficient Runtime We learn the local and global features for predicting the image-to-illumination mapping in low resolution, and perform bilateral grid based upsampling [8, 7, 12, 13] to enlarge the low-res prediction to the full resolution; see Figure 3.”
Face De-Spoofing: Anti-Spoofing via Noise Modeling (2018, methodology): “For de-mosaicing and super-resolution, many models, such as ResNet in [14,15] and joint models in [16,17,35], are learnt from the given pairs of low-quality input and high-quality ground truth.”
Learning to See in the Dark (2018, background): “Common choices for arranging raw data for a convolutional network are packing the color values into different channels with correspondingly lower spatial resolution, or duplicating and masking different colors [10].”
Learning Proximal Operators: Using Denoising Networks for Regularizing Inverse Imaging Problems (2017, methodology): “As we can see, the proposed method achieves a very high average PSNR value and is only surpassed by [15] who specifically trained a deep demosaicking CNN.”
A Generic Deep Architecture for Single Image Reflection Removal and Image Smoothing (2017, background): “Inspired by the tremendous success of deep learning for large-scale visual recognition tasks like ILSVRC [28, 20], a variety of recent work has investigated deep neural networks for low-level computer vision tasks such as image denoising [27, 11], shadow removal [15], and image smoothing [38, 24].”
Learning Enriched Features for Real Image Restoration and Enhancement (2020, result): “Recently, deep learning-based approaches [11,5,9,34,38,79,118,119] make signiﬁcant advances in image denoising, yielding favorable results than those of the hand-crafted methods.”
Deep Mean-Shift Priors for Image Restoration (2017, background): “RTF [15] Gharbi et al. [13] SEM [17] Ours”
Unprocessing Images for Learned Raw Denoising (2018, background): “In the modern era, most single-image denoising algorithms are entirely data-driven, consisting of deep neural networks trained to regress from noisy images to denoised images [15, 18, 32, 39, 42].”
Plug-and-Play Image Restoration With Deep Denoiser Prior (2020, background): “and the corresponding CFA pattern to recover a three-channel image [88], [89], [90].”
Fast End-to-End Trainable Guided Filter (2018, background): “ilter Recently, deep learning based methods are proposed in image processing tasks, which largely advanced the state-ofthe-art performance. Such tasks include image denoising [40], image demosaicking [41], image deblurring [42], image matting [43], rain drop removal [44], image dehazing [45], and im”
Deep bilateral learning for real-time image enhancement (2017, background): “…tasks such as depth estimation [Eigen et al. 2014], optical flow [Ilg et al. 2016], super-resolution [Dong et al. 2014], demosaicking and denoising [Gharbi et al. 2016; Zhang et al. 2016], image matting [Shen et al. 2016], colorization [Iizuka et al. 2016], and general image-to-image “translation”…”
Efficient and Explicit Modelling of Image Hierarchies for Image Restoration (2023, unknown): “Datasets Matlab DDR [75] DeepJoint [21] MMNet [36] RLDD [24] RNAN [85] DRUNet [80] GRL-S (ours)”
CycleISP: Real Image Restoration via Improved Data Synthesis (2020, methodology): “Several recent methods use deep CNNs [4, 7, 25, 28, 45, 64, 65, 2] and demonstrate promising denoising performance.”