Semantic Foggy Scene Understanding with Synthetic Data

By Christos Sakaridis, Dengxin Dai, Luc Van Gool. 2017

1,414 citations recorded by the source corpus.

This work addresses the problem of semantic foggy scene understanding (SFSU). Although extensive research has been performed on image dehazing and on semantic scene understanding with clear-weather images, little attention has been paid to SFSU. Due to the difficulty of collecting and annotating foggy images, we choose to generate synthetic fog on real images that depict clear-weather outdoor scenes, and then leverage these partially synthetic data for SFSU by employing state-of-the-art convolutional neural networks (CNN). In particular, a complete pipeline to add synthetic fog to real, clear-weather images using incomplete depth information is developed. We apply our fog synthesis on the Cityscapes dataset and generate Foggy Cityscapes with 20,550 images. SFSU is tackled in two ways: (1) with typical supervised learning, and (2) with a novel type of semi-supervised learning, which combines (1) with an unsupervised supervision transfer from clear-weather images to their synthetic foggy counterparts. In addition, we carefully study the usefulness of image dehazing for SFSU. For evaluation, we present Foggy Driving, a dataset with 101 real-world images depicting foggy driving scenes, which come with ground truth annotations for semantic segmentation and object detection. Extensive experiments show that (1) supervised learning with our synthetic data significantly improves the performance of state-of-the-art CNN for SFSU on Foggy Driving; (2) our semi-supervised learning strategy further improves performance; and (3) image dehazing marginally advances SFSU with our learning strategy. The datasets, models and code are made publicly available.

Subfields: Computer Vision and Pattern Recognition, Global and Planetary Change

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Global and Planetary Change

Citations in the corpus

Learning To Simulate (2018, methodology): “Consequently, prior approaches have used human knowledge to shape the generating distribution of the simulator (Sakaridis et al., 2018), or synthesized large-scale data with random parameters (Richter et al., 2016).”
Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming (2019, background): “Evaluating robustness to environmental changes in autonomous driving In recent years, weather conditions turned out to be a central limitation for state-of-the art autonomous driving systems [Sakaridis et al., 2018a, Volk et al., 2019, Dai and Van Gool, 2018, Chen et al., 2018, Lee et al., 2018].”
End-to-End Learning of Driving Models with Surround-View Cameras and Route Planners (2018, background): “Typical examples include the detection of roads [62], traffic lights [63], cars and pedestrians [64,65,1,2], and the tracking of such objects [66,67,68].”
Multi-Scale Boosted Dehazing Network With Dense Feature Fusion (2020, neutral): “, scene understanding [53] and detection [32]).”
Simulating Photo-realistic Snow and Fog on Existing Images for Enhanced CNN Training and Evaluation (2019, background): “sentable images [21], real fog affects every light wavelength (λ , [nm]) differently instead of applying the same effects on all three RGB channels.”
Gated Fusion Network for Single Image Dehazing (2018, background): “This is a classical image processing problem, which has received active research efforts in the vision communities since various high-level scene understanding tasks [19, 29, 32, 40] require the image dehazing to recover the clear scene.”
SoilingNet: Soiling Detection on Automotive Surround-View Cameras (2019, background): “[11] developed a robust semantic segmentation algorithm to handle foggy scenes.”
Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather (2019, background): “Both effects reduce contrast, and the observed foggy image can be modeled by [15] Ifoggy(x) = t(x)Iclear(x) + (1− t(x))L, (1) where Iclear is the latent clear image, depth-dependent transmissivity t, the global ambient component L.”
Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime (2018, background): “While perception algorithms based on visible light cameras are constantly getting better, they are mainly designed to operate on images taken at daytime under good illumination [21], [25].”
Unbiased Mean Teacher for Cross-domain Object Detection (2020, methodology): “The training set of the Cityscapes dataset is used as the source domain, and the Foggy Cityscapes dataset [38] is used as the target domain.”
Benchmarking Single-Image Dehazing and Beyond (2017, methodology): “Recently, [31] generated Foggy Cityscapes dataset with 20,550 images from the Cityscapes dataset, using incomplete depth information.”
Depth-Attentional Features for Single-Image Rain Removal (2019, methodology): “Then, we used a depth denoising method [37] to refine the depth maps of the picked images, and generated the rain streak intensity map tr(x) and fog layer A(x) from each depth map using Eqs.”
Guided Curriculum Model Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation (2019, background): “A case in point are recent efforts to adapt clear-weather models to fog [7, 26, 27], by using both labeled synthetic images and unlabeled real images of increasing fog density.”
Cooperative Semantic Segmentation and Image Restoration in Adverse Environmental Conditions (2019, methodology): “[36] drives from the Cityscapes and constitutes a collection of synthetic foggy images generated with a novel fog simu-”