Image De-Raining Using a Conditional Generative Adversarial Network

By He Zhang, Vishwanath A. Sindagi, Vishal M. Patel. 2017

1,193 citations recorded by the source corpus.

Severe weather conditions, such as rain and snow, adversely affect the visual quality of images captured under such conditions, thus rendering them useless for further usage and sharing. In addition, such degraded images drastically affect the performance of vision systems. Hence, it is important to address the problem of single image de-raining. However, the inherent ill-posed nature of the problem presents several challenges. We attempt to leverage powerful generative modeling capabilities of the recently introduced conditional generative adversarial networks (CGAN) by enforcing an additional constraint that the de-rained image must be indistinguishable from its corresponding ground truth clean image. The adversarial loss from GAN provides additional regularization and helps to achieve superior results. In addition to presenting a new approach to de-rain images, we introduce a new refined loss function and architectural novelties in the generator–discriminator pair for achieving improved results. The loss function is aimed at reducing artifacts introduced by GANs and ensure better visual quality. The generator sub-network is constructed using the recently introduced densely connected networks, whereas the discriminator is designed to leverage global and local information to decide if an image is real/fake. Based on this, we propose a novel single image de-raining method called image de-raining conditional generative adversarial network (ID-CGAN) that considers quantitative, visual, and also discriminative performance into the objective function. The experiments evaluated on synthetic and real images show that the proposed method outperforms many recent state-of-the-art single image de-raining methods in terms of quantitative and visual performances. Furthermore, the experimental results evaluated on object detection datasets using the Faster-RCNN also demonstrate the effectiveness of proposed method in improving the detection performance on images degraded by rain.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks (2017, background): “Compared with other deep generative models, Generative Adversarial Networks (GANs) [6] have shown great performance for generating sharper samples [17, 3, 23, 13, 10, 35, 24, 34, 39, 40].”
Black-Box vs. White-Box: Understanding Their Advantages and Weaknesses From a Practical Point of View (2019, background): “face frontal view generation [50], generating new human poses [51], photos to emojis [52], photograph editing [53]–[55], face aging [56], [57], photo blending [58], super resolution [59]–[61], photo inpainting [62]–[64], video prediction [65], and 3D object generation [66], [67]; among”
Photographic Text-to-Image Synthesis with a Hierarchically-Nested Adversarial Network (2018, methodology): “There are substantial existing methods investigating the better usage of GANs for different applications, such as image synthesis [32, 38], (unpaired) pixelto-pixel translation [14, 51], medical applications [4, 49], etc [20, 12, 52, 45].”
Pros and Cons of GAN Evaluation Measures (2018, background): “They can be utilized for purposes such as representation learning and semi-supervised learning [81, 76, 89], domain adaptation [27, 106], text to image synthesis [82], compression [103], super resolution [53], inpainting [80, 123], saliency prediction [79], image enhancement [127], style transfer an”
MedGAN: Medical Image Translation using GANs (2018, methodology): “The ﬁnal translation approach used in this comparative study is the ID-CGAN, designed for the denoising of weather artefacts in natural images (Zhang et al., 2017).”
HINet: Half Instance Normalization Network for Image Restoration (2021, methodology): “Test100 [61] Rain100H [49] Rain100L [49] Test2800 [10] Table 3: Deraining comparisons on Test100 [61], Rain100H [49], Rain100L [49], Test2800 [10] and Test1200 [60].”
Rain Removal in Traffic Surveillance: Does it Matter? (2018, methodology): “[3] both show a median improvement around 0%, with lower variability of the former method.”
Can deep learning beat numerical weather prediction? (2021, background): “such as image-to-image translation [49], super-resolution image generation [50], in-painting [51], image enhancement [52], image synthesis [53], style transfer and texture synthesis [54] and video generation and prediction [12,13].”
Crowd Counting via Adversarial Cross-Scale Consistency Pursuit (2018, methodology): “As suggested by previous works [9, 21, 35], we further include two conventional losses to smooth/improve the solution, which is detailed as follows.”
Advancing Image Understanding in Poor Visibility Environments: A Collective Benchmark Study (2019, methodology): “enhancement, HazeRD [60], OHAZE [61] and IHAZE [62] for dehazing, Rain100L/H [17] and Rain800 [63] for rain streak removal, and RAINDROP [64] for raindrop removal.”
Semi-Supervised Transfer Learning for Image Rain Removal (2018, methodology): “Zhang et al. [32] applied the mechanism of GAN and introduced a perceptual loss function for the consideration of rain removal problem.”
Densely Connected Pyramid Dehazing Network (2018, background): “The success of GANs in synthesizing realistic images has led researchers to explore the adversarial loss for various low-level vision applications such as text-to-image synthesis[32, 52, 54, 6], image-image translation [18, 28, 46] and other applications [23, 50, 55, 58, 45, 38, 44, 31].”
Uncertainty Guided Multi-Scale Residual Learning-Using a Cycle Spinning CNN for Single Image De-Raining (2019, methodology): “We conducted experiments on the real-world images provided by [34, 7, 35].”
Depth-Attentional Features for Single-Image Rain Removal (2019, methodology): “In the literature on single-image rain removal, existing methods focused on removing the rain streaks by adopting various image priors [5, 19, 22, 23, 31, 32, 41, 52], or by exploiting a deep convolutional neural network (CNN) to learn a mapping between the training images with and without rain stre”
Perceptual Adversarial Networks for Image-to-Image Transformation (2017, methodology): “For a fair comparison, we adopted the same image transformation network and data settings from ID-CGAN [24], and used the combination of different losses to perform the image de-raining (de-snowing) task.”
Density-Aware Single Image De-raining Using a Multi-stream Dense Network (2018, methodology): “Even though there exist several large-scale synthetic datasets [7, 41, 36], they lack the availability of the corresponding rain-density label information for each synthetic rainy image.”
The History Began from AlexNet: A Comprehensive Survey on Deep Learning Approaches (2018, methodology): “A single image de-raining system is proposed using a GAN recently [211].”
Jointly Optimize Data Augmentation and Network Training: Adversarial Data Augmentation in Human Pose Estimation (2018, background): “Generative Adversarial Networks (GANs) [12, 45, 47] includes two networks: generator and discriminator which compete against each other to improve performances.”