Dynamic Scene Deblurring Using Spatially Variant Recurrent Neural Networks

By Jiawei Zhang, Jin-shan Pan, Jimmy S. J. Ren, Yibing Song, Linchao Bao, Rynson W. H. Lau, Ming-Hsuan Yang. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

381 citations recorded by the source corpus.

Due to the spatially variant blur caused by camera shake and object motions under different scene depths, deblurring images captured from dynamic scenes is challenging. Although recent works based on deep neural networks have shown great progress on this problem, their models are usually large and computationally expensive. In this paper, we propose a novel spatially variant neural network to address the problem. The proposed network is composed of three deep convolutional neural networks (CNNs) and a recurrent neural network (RNN). RNN is used as a deconvolution operator performed on feature maps extracted from the input image by one of the CNNs. Another CNN is used to learn the weights for the RNN at every location. As a result, the RNN is spatially variant and could implicitly model the deblurring process with spatially variant kernels. The third CNN is used to reconstruct the final deblurred feature maps into restored image. The whole network is end-to-end trainable. Our analysis shows that the proposed network has a large receptive field even with a small model size. Quantitative and qualitative evaluations on public datasets demonstrate that the proposed method performs favorably against state-of-the-art algorithms in terms of accuracy, speed, and model size.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/79b0ee6a2aeca169786ab6a4952bbffd311757a8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Spatio-Temporal Filter Adaptive Network for Video Deblurring (2019, methodology): “Even though spatially variant RNNs are used in [44] to enlarge”
Bringing a Blurry Frame Alive at High Frame-Rate With an Event Camera (2018, methodology): “7, we show the quantitative and qualitative comparisons with the state-of-the-art image deblurring approaches [33, 22, 9, 11, 35, 40, 21], and the video reconstruction method [29] on our synthetic dataset, respectively.”
Real-World Blur Dataset for Learning and Benchmarking Deblurring Algorithms (2020, methodology): “We then benchmark state-of-the-art deblurring methods including both traditional optimization-based [46, 33, 18] and recent deep learning-based approaches [31, 49, 48, 21, 22, 42] using our test sets.”
Human-Aware Motion Deblurring (2019, background): “In a similar spirit, more advanced deep learning based deblurring models [35, 54, 27, 21, 58, 40] have been designed, e.”
An Underwater Image Enhancement Benchmark Dataset and Beyond (2019, methodology): “Data-driven methods: Recent years have witnessed significant advance of deep learning in low-level vision problems, including image super-resolution [46]–[48], image denoising [49], [50], image deblurring [51], [52], image dehazing [53], [54], etc.”
Enhancement of Underwater Images With Statistical Model of Background Light and Optimization of Transmission Map (2019, background): “There are many deep learning-based dehazing [37], debluring [38], denoising [30] methods for outdoor images.”
Neural Blind Deconvolution Using Deep Priors (2019, methodology): “Given the training set, deep CNNs can either be used to extract features to facilitate blur kernel estimation [1, 38], or be deployed to learn the direct mapping to clean image for motion de-blurring [7,25,43,51].”
Deblurring by Realistic Blurring (2020, background): “Zhang et al. [42] propose a spatially variant neural network, which includes three CNNs and one RNN. Tao et al. [36] propose an SRN-DeblurNet, which includes one LSTM and CNNs for multi-scale image deblurring.”
Uformer: A General U-Shaped Transformer for Image Restoration (2021, background): “[79], SRN [56], DMPHN [78], and MPRNet [77].”
Deep Stacked Hierarchical Multi-Patch Network for Image Deblurring (2019, background): “A network consisting of three deep CNNs and one RNN, proposed by [26], is a prominent example.”
Efficient Frequency Domain-based Transformers for High-Quality Image Deblurring (2022, background): “tectures, generative adversarial learning [8,9], physics model inspired network structures [14, 30], and so on.”
Dynamic Scene Deblurring With Parameter Selective Sharing and Nested Skip Connections (2019, methodology): “Recently, learning-based methods were proposed to facilitate the deblurring process by either replacing some steps in the traditional framework [31, 26, 5] or learning the end-to-end mapping from blurred to latent images [19, 33, 37].”
Artificial intelligence in the creative industries: a review (2020, methodology): “ck of nearby frames as input and uses synthetic motion blur to generate a training dataset. A Spatio-temporal recurrent network exploiting a dynamic temporal blending network is proposed in [115]. In [116], an encoder, recurrent network and decoder are concatenated to mitigate motion blur. Recently ”
UIEC^2-Net: CNN-based Underwater Image Enhancement Using Two Color Space (2021, background): “Unlike other low-level vision tasks such as image super-resolution [35], image denoising [36], image deblurring [37] and image de-hazing [38], which can use plenty of synthetic degraded images and high-quality counterparts for training, it is difﬁcult to synthesize realistic underwater images for…”
Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring (2020, neutral): “Method [24] [23] [5] [3] [13] [8] [20] [27] [2] [26] [9] Ours(a) Ours(b)”
Deep Image Deblurring: A Survey (2022, methodology): “GAN-based deep deblurringmethods have been applied to single image and video deblurring (Kupyn et al. 2018, 2019; Nah et al. 2017; Shen et al. 2018; Zhang et al. 2018).”