Burst Denoising with Kernel Prediction Networks

By Ben Mildenhall, Jonathan T. Barron, Jiawen Chen, Dillon Sharlet, Ren Ng, Robert E. Carroll. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017

434 citations recorded by the source corpus.

We present a technique for jointly denoising bursts of images taken from a handheld camera. In particular, we propose a convolutional neural network architecture for predicting spatially varying kernels that can both align and denoise frames, a synthetic data generation approach based on a realistic noise formation model, and an optimization guided by an annealed loss function to avoid undesirable local minima. Our model matches or outperforms the state-of-the-art across a wide range of noise levels on both real and synthetic data.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/84fc681efa7a7030098daf0c122ce6b770351c85

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Context-Gated Convolution (2019, methodology): “The family of work on dynamic filters [28,29,35] generates weights of convolution kernels using features extracted from input images by another CNN feature extractor.”
CARAFE: Content-Aware ReAssembly of FEatures (2019, background): “Within the realms of super-resolution and denoising, some other works [24, 14, 9] also explore using learnable kernels spatially in low-level vision.”
Omni-Dimensional Dynamic Convolution (2022, positive): “Kernel Prediction Networks (Bako et al., 2017; Mildenhall et al., 2018) leverage a CNN architecture to predict spatially varying kernels used for video denoising.”
Video Enhancement with Task-Oriented Flow (2017, positive): “…interpolation (Baker et al 2011; Werlberger et al 2011; Yu et al 2013; Jiang et al 2018; Sajjadi et al 2018), video super-resolution (Liu and Sun 2014; Liao et al 2015), and denoising (Liu and Free-man 2010; Varghese and Wang 2010; Maggioni et al 2012; Mildenhall et al 2018; Godard et al 2017).”
Seeing Motion in the Dark (2019, methodology): “Another option is to learn joint alignment and denoising [35], followed by subsequent pipeline processing (e.”
TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Up (2021, background): “Moreover, the spatial invariance possessed by convolution poses a bottleneck on its ability of adapting to spatially varying/heterogeneous visual patterns, which also motivates the success of relational network [28], dynamic filters [29, 30] and kernel prediction [31] methods.”
Toward Convolutional Blind Denoising of Real Photographs (2018, background): “However, as noted in [62, 41], taking both noisy image and noise level map as input is helpful in generalizing the learned model to images beyond the noise model and thus benefits blind denoising.”
NeRF in the Dark: High Dynamic Range View Synthesis from Noisy Raw Images (2021, background): “Subsequent work on both single [5, 9] and multi-image [8,18,36,49] denoising demonstrated the ben-eﬁts of training networks to operate directly on noisy raw input data.”
Deep Learning on Image Denoising: An overview (2019, methodology): “[95] CNN Impulse noise reduction CNN with PCA for image denoising Vogel et al. (2017) [187] CNN Gaussian image denoising U-net with multi scales technique for image denoising Mildenhall et al. (2018) [142] NN Low-light synthetic noisy image denoising, real noise Encoder-decoder with multi scales, an”
Unprocessing Images for Learned Raw Denoising (2018, background): “uirements of these large datasets of real photographs. Though it is generally understood that correctly modeling noise during image formation is critical for learning an effective denoising algorithm [20, 25, 27, 31], a less well-explored issue is the effect of the image processing pipeline used to ”
GridDehazeNet: Attention-Based Multi-Scale Network for Image Dehazing (2019, background): “The grid network has clear advantages over the encoderdecoder network and the conventional multi-scale network extensively used in image restoration [18, 41, 38, 27].”