Model Adaptation with Synthetic and Real Data for Semantic Dense Foggy Scene Understanding

By Christos Sakaridis, Dengxin Dai, Simon Hecker, Luc Van Gool. 2018

289 citations recorded by the source corpus.

This work addresses the problem of semantic scene understanding under dense fog. Although considerable progress has been made in semantic scene understanding, it is mainly related to clear-weather scenes. Extending recognition methods to adverse weather conditions such as fog is crucial for outdoor applications. In this paper, we propose a novel method, named Curriculum Model Adaptation (CMAda), which gradually adapts a semantic segmentation model from light synthetic fog to dense real fog in multiple steps, using both synthetic and real foggy data. In addition, we present three other main stand-alone contributions: (1) a novel method to add synthetic fog to real, clear-weather scenes using semantic input; (2) a new fog density estimator; (3) the Foggy Zurich dataset comprising 3808 real foggy images, with pixel-level semantic annotations for 16 images with dense fog. Our experiments show that (1) our fog simulation slightly outperforms a state-of-the-art competing simulation with respect to the task of semantic foggy scene understanding (SFSU); (2) CMAda improves the performance of state-of-the-art models for SFSU significantly by leveraging unlabeled real foggy data. The datasets and code will be made publicly available.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Benchmarking Robustness in Object Detection: Autonomous Driving when Winter is Coming (2019, background): “Evaluating robustness to environmental changes in autonomous driving In recent years, weather conditions turned out to be a central limitation for state-of-the art autonomous driving systems [Sakaridis et al., 2018a, Volk et al., 2019, Dai and Van Gool, 2018, Chen et al., 2018, Lee et al., 2018].”
You Only Look Yourself: Unsupervised and Untrained Single Image Dehazing Neural Network (2020, unknown): “Therefore, image dehazing, as a preprocessing step and visual enhancement technology, has been extensively researched and achieves remarkable performance (Cai et al., 2016; Li et al., 2017, 2015; Sakaridis et al., 2018a,b; Tan, 2008; Zhang and Patel, 2018; Zhu et al., 2018).”
CrDoCo: Pixel-Level Domain Transfer With Cross-Domain Consistency (2019, methodology): “Training the model on large-scale synthetic datasets has been extensively studied in semantic segmentation [41, 42, 15, 14, 7, 17, 34, 35, 53], multi-view stereo [18], depth estimation [48], optical flow [40, 19, 21], amodal segmentation [16], and object detection [7, 30].”
DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation (2021, methodology): “Most of the UDA methods pre-compute the pseudo-labels offline, train the model, and repeat the process [13,84,92,93].”
Guided Curriculum Model Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation (2019, methodology): “A case in point are recent efforts to adapt clear-weather models to fog [7, 26, 27], by using both labeled synthetic images and unlabeled real images of increasing fog density.”
Attention Guided Low-Light Image Enhancement with a Large Scale Low-Light Simulation Dataset (2019, methodology): “Using synthetic data allows easy model adaptation for target conditions without requiring additional manual annotations [17,63].”
Contrastive Learning for Compact Single Image Dehazing (2021, background): “put images captured under hazy scenes significantly affect the performance of high-level computer vision tasks, such as object detection [26, 8] and scene understanding [39, 40].”
MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation (2022, methodology): “S5 further provides the semantic segmentation performance of MIC on Cityscapes [11] to Foggy Zurich [74].”
Dark Model Adaptation: Semantic Image Segmentation from Daytime to Nighttime (2018, background): “The concurrent work in [24] on adaptation of semantic models from clear weather condition to light fog then to dense fog is closely related to ours.”
NeurAll: Towards a Unified Visual Perception Model for Automated Driving (2019, methodology): “An example is the work of Itu et al. [21] which uses CNN to estimate vanishing point for obtaining extrinsic calibration, and Sakaridis et al. [22] on foggy scene understanding.”
ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding (2021, background): “CS: Cityscapes [8], FC: Foggy Cityscapes [38], FC-DBF: Foggy Cityscapes-DBF [37], FZ: Foggy Zurich [37], ND: Nighttime Driving [10], DZ: Dark Zurich [40].”
Automatic Adaptation of Object Detectors to New Domains Using Self-Training (2019, background): “Similarly motivated approaches have been applied to adapting to foggy street scenes in a progressive manner [47, 10].”
Unsupervised Intra-Domain Adaptation for Semantic Segmentation Through Self-Supervision (2020, background): “For curriculum domain adaptation on foggy scene understanding, [22] proposes to adapt a semantic segmentation model from non-foggy images to synthetic light foggy images, and then to real heavy foggy images.”
Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather (2019, result): “ observed foggy image can be modeled as I foggy(x) = t(x)I clear(x)+(1 t(x))L; (1) where I clear is the latent clear image, depth-dependent transmissivity t, the global ambient component L, following [43,45]. The transmission coefﬁcient t(x) = exp( d(x)), where is the fog density or attenuation coef”
Open Compound Domain Adaptation (2019, methodology): “Unlike existing curriculum adaptation methods [54, 5, 27, 23, 56, 55] that rely on some holistic measure of instance difficulty, we schedule the learning of unlabeled instances in the compound target domain according to their individual gaps to the labeled source domain, so that we solve an incremen”
NeurAll: Towards a Unified Model for Visual Perception in Automated Driving (2019, unknown): “[22] on foggy scene understanding.”