Deep Joint Rain Detection and Removal from a Single Image

By Wenhan Yang, Robby T. Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, Shuicheng Yan. 2016

1,283 citations recorded by the source corpus.

In this paper, we address a rain removal problem from a single image, even in the presence of heavy rain and rain streak accumulation. Our core ideas lie in our new rain image model and new deep learning architecture. We add a binary map that provides rain streak locations to an existing model, which comprises a rain streak layer and a background layer. We create a model consisting of a component representing rain streak accumulation (where individual streaks cannot be seen, and thus visually similar to mist or fog), and another component representing various shapes and directions of overlapping rain streaks, which usually happen in heavy rain. Based on the model, we develop a multi-task deep learning architecture that learns the binary rain streak map, the appearance of rain streaks, and the clean background, which is our ultimate output. The additional binary map is critically beneficial, since its loss function can provide additional strong information to the network. To handle rain streak accumulation (again, a phenomenon visually similar to mist or fog) and various shapes and directions of overlapping rain streaks, we propose a recurrent rain detection and removal network that removes rain streaks and clears up the rain accumulation iteratively and progressively. In each recurrence of our method, a new contextualized dilated network is developed to exploit regional contextual information and to produce better representations for rain detection. The evaluation on real images, particularly on heavy rain, shows the effectiveness of our models and architecture.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Safety, Risk, Reliability and Quality

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

In Safety, Risk, Reliability and Quality

Citations in the corpus

Memory-Efficient Hierarchical Neural Architecture Search for Image Denoising (2019, neutral): “7763 JORDER-R [39] 22.”
CLEARER: Multi-Scale Neural Architecture Search for Image Restoration (2020, methodology): “From the left to the right are Input, DSC [26], LP [18], DetailsNet [6], JORDER [45], SCAN [17], RESCAN [17], CLEARER, and Ground truth.”
Recurrent Squeeze-and-Excitation Context Aggregation Net for Single Image Deraining (2018, methodology): “[9,26] design a deep recurrent dilated network to jointly detect and remove rain streaks.”
Uncertainty Guided Multi-Scale Residual Learning-Using a Cycle Spinning CNN for Single Image De-Raining (2019, methodology): “[31] for testing UMRL based cycle spinning method.”
Depth-Attentional Features for Single-Image Rain Removal (2019, methodology): “Table 3: Comparison with the state-of-the-arts using the PSNR and SSIM on the Rain100H dataset [47].”
Low-Light Image and Video Enhancement Using Deep Learning: A Survey (2021, methodology): “The commonly used loss functions in LLIE networks are also employed in image reconstruction networks for image super-resolution [73], image denoising [74], image detraining [75], [76], [77], and image deblurring [78].”
Human-Aware Motion Deblurring (2019, methodology): “Numerous CNN-based methods have also been applied in an endto-end fashion for image processing and generation problems, such as segmentation [24, 42], super-resolution [53, 22, 14, 15, 44], denoising [25, 49, 59, 47, 48], dehazing and deraining [32, 55], enhancement [31, 56] etc.”
Multi-Scale Progressive Fusion Network for Single Image Deraining (2020, methodology): “There is no uniﬁed training datasets for all competing meth-ods in this paper, e.g. PreNet refers to JORDER [35] and uses 1254 pairs for training.”
Advancing Image Understanding in Poor Visibility Environments: A Collective Benchmark Study (2019, methodology): “enhancement, HazeRD [60], OHAZE [61] and IHAZE [62] for dehazing, Rain100L/H [17] and Rain800 [63] for rain streak removal, and RAINDROP [64] for raindrop removal.”
Joint Bi-layer Optimization for Single-Image Rain Streak Removal (2017, methodology): “[32] independently presented two different deep models for single-image rain streak removal.”
Image De-Raining Using a Conditional Generative Adversarial Network (2017, methodology): “• JORDER: CNN-based method [18] (CVPR ’17)”
Restoring Vision in Adverse Weather Conditions With Patch-Based Denoising Diffusion Models (2022, methodology): “deep detail network [2], [41], and the joint rain detection and removal (JORDER) method [42].”
Density-Aware Single Image De-raining Using a Multi-stream Dense Network (2018, methodology): “Similar to [36], we also leverage a multi-stream network to capture the rain-streak components with different scales and shapes.”
HINet: Half Instance Normalization Network for Image Restoration (2021, methodology): “Datasets As in [56], we train our models on SIDD [1] for image denoising, GoPro [31] for image deblurring, and 13,712 clean-rain image pairs (for simplicity, denoted as Rain13k in the following) gathered from [10, 25, 49, 60, 61] for image deraining.”
Single Image Deraining: A Comprehensive Benchmark Analysis (2019, methodology): “Degraded GMM [27] JORDER [32] DDN [6] CGAN [33] DID-MDN [8] DeRaindrop [7]”
Attentive Generative Adversarial Network for Raindrop Removal from A Single Image (2017, background): “There are a few papers dealing with bad weather visibility enhancement, which mostly deal with haze or fog (e.g. [19, 7, 16]), and rain streaks (e.g. [3, 2, 14, 24]).”
IR-NAS: Neural Architecture Search for Image Restoration (2019, background): “[12] design a deep recurrent dilated network to jointly detect and remove rain streaks.”