Deep Multi-scale Convolutional Neural Network for Dynamic Scene Deblurring

By Seungjun Nah, Tae Hyun Kim, Kyoung Mu Lee. Computer Vision and Pattern Recognition, 2016

2,495 citations recorded by the source corpus.

Non-uniform blind deblurring for general dynamic scenes is a challenging computer vision problem as blurs arise not only from multiple object motions but also from camera shake, scene depth variation. To remove these complicated motion blurs, conventional energy optimization based methods rely on simple assumptions such that blur kernel is partially uniform or locally linear. Moreover, recent machine learning based methods also depend on synthetic blur datasets generated under these assumptions. This makes conventional deblurring methods fail to remove blurs where blur kernel is difficult to approximate or parameterize (e.g. object motion boundaries). In this work, we propose a multi-scale convolutional neural network that restores sharp images in an end-to-end manner where blur is caused by various sources. Together, we present multi-scale loss function that mimics conventional coarse-to-fine approaches. Furthermore, we propose a new large-scale dataset that provides pairs of realistic blurry image and the corresponding ground truth sharp image that are obtained by a high-speed camera. With the proposed model trained on this dataset, we demonstrate empirically that our method achieves the state-of-the-art performance in dynamic scene deblurring not only qualitatively, but also quantitatively.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/65b16da51891a6b98140d425804c8a0fd0299219

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Uformer: A General U-Shaped Transformer for Image Restoration (2021, methodology): “[42], De-blurGAN [27], Xu et al .”
Deep Generalized Unfolding Networks for Image Restoration (2022, methodology): “For image deblurring, similar to [36, 55, 63, 79, 80], we train our model with 2, 103 image pairs from GoPro [47] dataset and evaluate each method on the test sets from GoPro and HIDE [58], which contain 1, 111 and 2, 025 samples, respectively.”
HINet: Half Instance Normalization Network for Image Restoration (2021, methodology): “In addition, we conduct experiments on GoPro [31] dataset for image deblurring following [56, 57] etc.”
DiffIR: Efficient Diffusion Model for Image Restoration (2023, methodology): “We train DiffIR on GoPro [45] dataset for image motion deblurring and evaluate DiffIR on two classic benchmarks (GoPro, HIDE [53]).”
GridDehazeNet: Attention-Based Multi-Scale Network for Image Dehazing (2019, methodology): “Following [19, 15], we do not use batch normalization.”
OmniGen: Unified Image Generation (2024, neutral): “A man and a short-haired woman with a wrinkled face are standing in front of a bookshelf in a library; the man is the man in the middle of |image_1|, and the woman is the oldest in |image_2| A man and a woman are sitting at a classroom desk.”
Image Restoration with Mean-Reverting Stochastic Differential Equations (2023, methodology): “, 2019), deblurring (Nah et al., 2017; Zhang et al., 2020), denoising (Zhang et al.”
DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks (2017, methodology): “The first model to which we refer as DeblurGANWILD was trained on a random crops of size 256x256 from 1000 GoPro training dataset images [25] downscaled by a factor of two.”
Algorithm Unrolling: Interpretable, Efficient Deep Learning for Signal and Image Processing (2019, background): “ere the tissue is low rank and the blood vessels are sparse. They then unroll an ISTA approach to robust PCA into a deep network, called Convo12 (a) Groundtruth (b) Perrone et al. [49] (c) Nah et al. [50] (d) Tao et al. [51] (e) DUBLID Fig. 9. Sample experimental results from [40] for visual compari”
Deep Network Interpolation for Continuous Imagery Effect Transition (2018, background): “CNN-based approaches have led to a series of breakthroughs for several image restoration tasks including super-resolution [3, 18, 25, 19, 33, 20, 43], denoising [1, 44], de-blocking [40, 7] and deblurring [42, 32, 28].”
Scaling Up to Excellence: Practicing Model Scaling for Photo-Realistic Image Restoration In the Wild (2024, background): “In the early stage, researchers independently explored different types of image degradation, such as super-resolution (SR) [13, 19, 20], denoising [11, 87, 89], and deblurring [14, 58, 70].”
Spatio-Temporal Filter Adaptive Network for Video Deblurring (2019, methodology): “More recently, many end-to-end CNN models [38, 44, 17, 23, 26] have also been proposed for image deblurring.”
Dynamic Scene Deblurring Using Spatially Variant Recurrent Neural Networks (2018, methodology): “[21] to approximate this problem) do not handle this problem well.”
Deblurring via Stochastic Refinement (2021, methodology): “This was done to make qualitative evaluation in a more controlled environment, since the low-quality ground truth images in existing paired datasets [50, 56] make qualitative assessment difﬁcult.”
VRT: A Video Restoration Transformer (2022, methodology): “C. Video Deblurring 1) Quantitative Results: We conducts experiments on three different datasets for fair comparison with existing meth-ods [2], [4], [5], [6], [16], [22], [24], [36], [62], [64].”
Scale-Recurrent Network for Deep Image Deblurring (2018, methodology): “Our experiments show that with the recurrent structure and combining above advantages, our end-to-end deep image deblurring framework can greatly improve training ef-ﬁciency ( ≈ 1/4 training time of [25] to accomplish similar restoration).”
Efficient and Explicit Modelling of Image Hierarchies for Image Restoration (2023, background): “GoPro dataset [51] is used for training.”
Channel Attention Is All You Need for Video Frame Interpolation (2020, methodology): “So, we construct our benchmark dataset based on high frame rate videos including 11 videos from the test set of GOPRO dataset (Nah, Kim, and Lee 2017) and 20 videos collected from YouTube.”