Deep Stacked Hierarchical Multi-Patch Network for Image Deblurring

By Hongguang Zhang, Yuchao Dai, Hongdong Li, Piotr Koniusz. Computer Vision and Pattern Recognition, 2019

649 citations recorded by the source corpus.

Despite deep end-to-end learning methods have shown their superiority in removing non-uniform motion blur, there still exist major challenges with the current multi-scale and scale-recurrent models: 1) Deconvolution/upsampling operations in the coarse-to-fine scheme result in expensive runtime; 2) Simply increasing the model depth with finer-scale levels cannot improve the quality of deblurring. To tackle the above problems, we present a deep {hierarchical multi-patch network} inspired by Spatial Pyramid Matching to deal with blurry images via a fine-to-coarse hierarchical representation. To deal with the performance saturation w.r.t. depth, we propose a stacked version of our multi-patch model. Our proposed basic multi-patch model achieves the state-of-the-art performance on the GoPro dataset while enjoying a 40$\times$ faster runtime compared to current multi-scale methods. With 30ms to process an image at 1280$\times$720 resolution, it is the first real-time deep motion deblurring model for 720p images at 30fps. For stacked networks, significant improvements (over 1.2dB) are achieved on the GoPro dataset by increasing the network depth. Moreover, by varying the depth of the stacked model, one can adapt the performance and runtime of the same network for different application scenarios.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/a5c1156c2c8185df4581cf139df05aab66b3bb22

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Deep Generalized Unfolding Networks for Image Restoration (2022, methodology): “We compare our method with several competitive works [50, 61, 80, 90] and the recent best algorithm MPRNet [79].”
Uformer: A General U-Shaped Transformer for Image Restoration (2021, methodology): “Following previous methods [74,75], we train Uformer only on the GoPro dataset [42], and evaluate it on the test set of GoPro, HIDE [49], and RealBlur-R/-”
Spatially-Attentive Patch-Hierarchical Network for Adaptive Motion Deblurring (2020, methodology): “Note that unlike [26], we also avoid cascading of our network along depth, as that adds severe computational burden.”
Stripformer: Strip Transformer for Fast Image Deblurring (2022, methodology): “The deblurred results from left to right are produced by DSD [6], DMPHN [9], MTRNN [10], MIMO [42], MPRNet [33] and our method, respectively. fine-grained information like facial expressions on the HIDE dataset.”
Blind Face Restoration via Deep Multi-scale Component Dictionaries (2020, background): “, image super-resolution [9,19,24,44, 46], denoising [13, 38, 42, 43], deblurring [22, 29, 41], and compression artifact removal [8, 10, 12].”
Plug-and-Play Image Restoration With Deep Denoiser Prior (2020, methodology): “To evaluate the effectiveness of the proposed DPIR, we choose four representative methods for comparison, including model-based method EPLL [4], learning-based nonblind method FDN [77], learning-based blind method DMPHN [78] and plug-and-play method IRCNN and IRCNN+.”
Multi-Stage Progressive Image Restoration (2021, methodology): “Third, unlike in [88], it is important to provide ground-truth supervision at each stage for progressive restoration.”
Efficient Frequency Domain-based Transformers for High-Quality Image Deblurring (2022, methodology): “…main success of these methods is due to developing kinds of network architectural designs, for example, the multi-scale [4,13,19] or multi-stage [28,29] Comparisons of the proposed method and state-of-the-art ones on the GoPro dataset [13] in terms of accuracy, ﬂoating point operations (FLOPs),…”
Deblurring via Stochastic Refinement (2021, methodology): “For the ease of interpretation, we used negative Kernel Inception Distance [4] ( C − KID for a constant C ) as the measure of perceptual quality. deblurring models [14,36,41,61,65,67,68,76].”
Ensemble deep learning: A review (2021, methodology): “2019 Deep stacked hierarchical multi-patch network for image deblurring [91] Stacking Deblurring Image”
Rethinking Coarse-to-Fine Approach in Single Image Deblurring (2021, methodology): “We compared MIMO-UNet with state-of-the-art deblurring networks [20, 31, 5, 35, 29, 22, 23, 33, 28].”
MAXIM: Multi-Axis MLP for Image Processing (2022, methodology): “Recently, multi-stage networks [15, 33, 108, 110] have attained promising results relative to the aforementioned single-stage models on the challenging deblurring and deraining tasks [22, 33, 108].”
Restormer: Efficient Transformer for High-Resolution Image Restoration (2021, neutral): “DMPHN (CVPR19) DBGAN (CVPR20) [94] [58] [14] [13]”
HINet: Half Instance Normalization Network for Image Restoration (2021, methodology): “For example, with the help of HIN, DMPHN [57] increased 0.”
A comprehensive review on ensemble deep learning: Opportunities and challenges (2023, methodology): “Also, in Zhang et al. (2019) applied of stacking method based on multiple CNNs used for Image Deblurring.”
Deep Image Deblurring: A Survey (2022, methodology): “Based on this loss function, Zhang et al. (2020) replace the standard adversarial loss with the relativistic loss:”
VRT: A Video Restoration Transformer (2022, methodology): “Method DeepDeblur [66] SRN [89] DMPHN [113] SAPHN [83] MPRNet [112] SFE [104] IFI-RNN [67] ESTRNN [122] EDVR [99] TSP [71] PVDNet [80] GSTA [84] VRT (ours)”
Pre-Trained Image Processing Transformer (2020, methodology): “Method MSCNN [54] SRN [67] DSD [30] DeblurGANv2 [44] DMPHN [84] LEBMD [40] EDSD [81]”