Multi-Scale Progressive Fusion Network for Single Image Deraining

By Kui Jiang, Zhongyuan Wang, Peng Yi, Chen Chen, Baojin Huang, Yimin Luo, Jiayi Ma, Junjun Jiang. Computer Vision and Pattern Recognition, 2020

826 citations recorded by the source corpus.

Rain streaks in the air appear in various blurring degrees and resolutions due to different distances from their positions to the camera. Similar rain patterns are visible in a rain image as well as its multi-scale (or multi-resolution) versions, which makes it possible to exploit such complementary information for rain streak representation. In this work, we explore the multi-scale collaborative representation for rain streaks from the perspective of input image scales and hierarchical deep features in a unified framework, termed multi-scale progressive fusion network (MSPFN) for single image rain streak removal. For the similar rain streaks at different positions, we employ recurrent calculation to capture the global texture, thus allowing to explore the complementary and redundant information at the spatial dimension to characterize target rain streaks. Besides, we construct multi-scale pyramid structure, and further introduce the attention mechanism to guide the fine fusion of these correlated information from different scales. This multi-scale progressive fusion strategy not only promotes the cooperative representation, but also boosts the end-to-end training. Our proposed method is extensively evaluated on several benchmark datasets and achieves the state-of-the-art results. Moreover, we conduct experiments on joint deraining, detection, and segmentation tasks, and inspire a new research direction of vision task driven image deraining. The source code is available at https://github.com/kuihua/MSPFN.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/1234b5451347ce6fe3bae9c34e67edbacaf73b52

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Learning A Sparse Transformer Network for Effective Image Deraining (2023, methodology): “To better represent the rain distribution, several studies take rain characteristics such as rain direction [27], density [61], veiling effect [15] into account, and optimize the network structure via recursive computation [19, 23, 36] or transfer mechanism [16, 49, 54, 55].”
Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models (2022, methodology): “Novel solutions introduced hierarchical multi-scale feature extraction and fusion [52], as well as its extension progressive coupled networks (PCNet) [53] which were shown to outperform several methods on combined deraining and dehazing tasks.”
Multi-Stage Progressive Image Restoration (2021, methodology): “Compared to the recent best algorithm MSPFN [36], we obtain a performance gain of 1 .”
HINet: Half Instance Normalization Network for Image Restoration (2021, methodology): “Rain13k For image deraining, we train our model on Rain13k as described above and evaluate results by Y channel in YCbCr color space following [19, 56].”
Restormer: Efficient Transformer for High-Resolution Image Restoration (2021, methodology): “We compute PSNR/SSIM scores using the Y channel in YCbCr color space in a way similar to existing methods [32, 61, 93].”
Image De-raining via Continual Learning (2021, background): “In recent years, remarkable progress has been achieved on single image de-raining and other low-level vision tasks due to the rapid growth of deep learning [32, 9, 5, 35, 45, 18, 50, 43, 27, 8, 44, 41, 33, 23, 13, 37, 3, 6, 30].”
Image De-Raining Transformer (2022, methodology): “…for image de-raining, various loss functions are developed, from single MSE loss [69], l 1 loss [21], negative SSIM loss [72] to more elaborated loss functions, e.g., hybrid l 1 þ SSIM [80], attention loss [73], adversarial loss [63], edge loss [59], Charbonnier loss [79] and perceptual loss [76].”
All-In-One Image Restoration for Unknown Corruption (2022, background): “To be specific, the denoising baselines contain CBM3D [5], DnCNN [53], IRCNN [54], FFDNet [55] and BRDNet [40]. the deraining baselines are DIDMDN [52], UMRL [50], SIRR [46], MSPFN [17], and LPNet [11].”
PSCC-Net: Progressive Spatio-Channel Correlation Network for Image Manipulation Detection and Localization (2021, background): “It has been widely adopted in many low-level and high-level vision tasks, such as denoising [5], [49], inpainting [50], super-resolution [51], [52], and object detection [53]–[56].”
CLEARER: Multi-Scale Neural Architecture Search for Image Restoration (2020, background): “Although it is well-known that multi-resolution fusion is helpful to boosting image recovery performance [7, 12, 44, 23], it is difficult to obtain an effective architecture and almost all existing works rely on human design.”
MAXIM: Multi-Axis MLP for Image Processing (2022, methodology): “Visual examples for image deraining on Rain100L [102] among RESCAN [48], PreNet [75], MSPFN [33], MPRNet [108], HINet [15], and our MAXIM-2S model.”
Deep Generalized Unfolding Networks for Image Restoration (2022, methodology): “We compare our proposed DGUNet with eight competitive methods [19,31,38,54,68,72,79,81].”
Image Inpainting by End-to-End Cascaded Refinement With Mask Awareness (2021, background): “There are also other frameworks proposed for many other low-level tasks, such as image rain removal [42]–[44] and super-resolution [45]–[47].”
Cross-level Feature Aggregation Network for Polyp Segmentation (2023, background): “Jiang et al. [26] utilized a multi-scale progressive fusion module to fully exploit the inherent correlations among multi-scale rain streaks.”