Guided Curriculum Model Adaptation and Uncertainty-Aware Evaluation for Semantic Nighttime Image Segmentation

By Christos Sakaridis, Dengxin Dai, Luc Van Gool. IEEE International Conference on Computer Vision, 2019

306 citations recorded by the source corpus.

Most progress in semantic segmentation reports on daytime images taken under favorable illumination conditions. We instead address the problem of semantic segmentation of nighttime images and improve the state-of-the-art, by adapting daytime models to nighttime without using nighttime annotations. Moreover, we design a new evaluation framework to address the substantial uncertainty of semantics in nighttime images. Our central contributions are: 1) a curriculum framework to gradually adapt semantic segmentation models from day to night via labeled synthetic images and unlabeled real images, both for progressively darker times of day, which exploits cross-time-of-day correspondences for the real images to guide the inference of their labels; 2) a novel uncertainty-aware annotation and evaluation framework and metric for semantic segmentation, designed for adverse conditions and including image regions beyond human recognition capability in the evaluation in a principled fashion; 3) the Dark Zurich dataset, which comprises 2416 unlabeled nighttime and 2920 unlabeled twilight images with correspondences to their daytime counterparts plus a set of 151 nighttime images with fine pixel-level annotations created with our protocol, which serves as a first benchmark to perform our novel evaluation. Experiments show that our guided curriculum adaptation significantly outperforms state-of-the-art methods on real nighttime sets both for standard metrics and our uncertainty-aware metric. Furthermore, our uncertainty-aware evaluation reveals that selective invalidation of predictions can lead to better results on data with ambiguous content such as our nighttime benchmark and profit safety-oriented applications which involve invalid inputs.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/5e4bc76fb2a00151e43aee7fd2f49ef527486634

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dengxin Dai
3. Jiaying Liu
4. Ming-Hsuan Yang
5. Feng Lu
6. Chen Change Loy
7. Yu Li
8. Wenhan Yang

Citations in the corpus

• Attention Guided Low-Light Image Enhancement with a Large Scale Low-Light Simulation Dataset (2019, background): “Recently, deep learning has achieved great success in the field of low-level image processing [66] and nighttime scenes modeling [57] and understanding [18,62].”
• Benchmarking Low-Light Image Enhancement and Beyond (2021, background): “In Dai and Gool (2018), Sakaridis et al. (2019), two datasets including unlabeled nighttime images, unlabeled twilight images with correspondences to their daytime versions, and night-time images with pixel-level dense annotations are proposed to serve evaluation of semantic segmentation at night.”
• Is Context-Aware CNN Ready for the Surroundings? Panoramic Semantic Segmentation in the Wild (2021, background): “Overall, WildPASS has more than 111.5 million annotated pixels for 500 panoramas, which is larger than state-of-the-art evaluation-oriented datasets including WildDash [60], DarkZurich [50], RainyNight [61], ISSAFE [62] and the previous PASS dataset [9].”
• Toward Fast, Flexible, and Robust Low-Light Image Enhancement (2022, unknown): “Low-light image enhancement aims at making information hidden in the dark visible to improve image quality, it has drawn much attention in multiple emerging computer vision areas [18, 24, 25] recently.”
• ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding (2021, background): “…adaptation in their experiments [10,11,32,37,38,39,40] and whenever they do, they either restrict the target adverse domain to a single condition, e.g. nighttime [10, 39, 40], fog [37, 38], or rain [11], or do not include a quantitative evaluation on the real target domain altogether [32].”
• MetaCorrection: Domain-aware Meta Loss Correction for Unsupervised Domain Adaptation in Semantic Segmentation (2021, unknown): “The general idea of UDA semantic segmentation methods is to perform domain alignment through adversarial learning [2, 6, 7, 9, 11, 20, 22, 26, 40, 41] or utilize self-training strategy on target samples to adapt the segmentation models [4, 17, 23, 28, 30, 35, 46, 48, 49, 50, 51].”
• Seeing Through Fog Without Seeing Fog: Deep Multimodal Sensor Fusion in Unseen Adverse Weather (2019, background): “Domain Adaptation Another line of research tackles the shift of unlabeled data distributions by domain adaptation [61, 29, 51, 28, 70, 63].”