Learning to Adapt Structured Output Space for Semantic Segmentation

By Yi-Hsuan Tsai, Wei-Chih Hung, Samuel Schulter, Kihyuk Sohn, Ming-Hsuan Yang, Manmohan Chandraker. 2018

1,735 citations recorded by the source corpus.

Convolutional neural network-based approaches for semantic segmentation rely on supervision with pixel-level ground truth, but may not generalize well to unseen image domains. As the labeling process is tedious and labor intensive, developing algorithms that can adapt source ground truth labels to the target domain is of great interest. In this paper, we propose an adversarial learning method for domain adaptation in the context of semantic segmentation. Considering semantic segmentations as structured outputs that contain spatial similarities between the source and target domains, we adopt adversarial learning in the output space. To further enhance the adapted model, we construct a multi-level adversarial network to effectively perform output space domain adaptation at different feature levels. Extensive experiments and ablation study are conducted under various domain adaptation settings, including synthetic-to-real and cross-city scenarios. We show that the proposed method performs favorably against the state-of-the-art methods in terms of accuracy and visual quality.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Domain Adaptation for Traffic Density Estimation (2021, methodology): “Another interesting work that inspired this paper is (Tsai et al., 2018), where the authors applied adversarial training to the output space taking advantage of the structural consistency across domains.”
Learning To Simulate (2018, background): “Using simulated data for semantic segmentation was recently investigated from a domain adaptation perspective (Tsai et al., 2018; Richter et al., 2016), where an abundant set of simulated data is leveraged to train models applicable on real data.”
Unsupervised Model Adaptation for Continual Semantic Segmentation (2020, neutral): “We have compared our performance against the adversarial learning-based UDA methods: GIO-Ada (Chen et al. 2019b), ADVENT (Vu et al. 2019), AdaSegNet (Tsai et al. 2018), TGCF-DA+SE (Choi, Kim, and Kim 2019), PCEDA (Yang et al. 2020), and CyCADA (Hoffman et al. 2018a).”
ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation (2018, methodology): “To further improve performance on ResNet-101, we perform adaptation on multi-level outputs coming from both conv4 and conv5 features [41].”
Instance Adaptive Self-Training for Unsupervised Domain Adaptation (2020, methodology): “uilding shared embedding space to both the source and target domain. Following the same idea, many adversarial training based UDA methods are proposed by adding a domain discriminator in recent years [27,10,21,6,29,33,32]. With adversarial training, the domain adversarial loss can be minimized to di”
Self-Ensembling With GAN-Based Data Augmentation for Domain Adaptation in Semantic Segmentation (2019, methodology): “We compare our proposed method with Curriculum DA [51], CyCADA [14], MCD [39], LSD-seg [41], AdaptSegNet [44], ROAD [5], Conservative Loss [55], DCAN [47], and CBST [57].”
Open Compound Domain Adaptation (2019, methodology): “Following [47, 58, 35], a pre-trained VGG-16 [44] is the backbone network for CDriving.”
Continual Test-Time Domain Adaptation (2022, methodology): “During training, UDA methods often align the feature distributions between the two domains using discrepancy losses [39] or adversarial training [12, 58].”
Bidirectional Learning for Domain Adaptation of Semantic Segmentation (2019, methodology): “But we can still achieve at least 4% better than most of other results given by [37, 39, 36, 33].”
Taking a Closer Look at Domain Shift: Category-Level Adversaries for Semantics Consistent Domain Adaptation (2018, methodology): “We present the adaptation results on task GTA5 → Cityscapes in Table 1 with comparisons to the state-of-the-art domain adaptation methods [35, 45, 15, 14, 41, 50].”
Diverse Image-to-Image Translation via Disentangled Representations (2018, background): “Domain Adversarial Neural Network (DANN) [11,13] and its variants [38,4,37] tackle domain adaptation through learning domain-invariant features.”
Permuted AdaIN: Reducing the Bias Towards Global Statistics in Image Classification (2020, neutral): “AdaptSegNet [38] 86.”
ESL: Entropy-guided Self-supervised Learning for Domain Adaptation in Semantic Segmentation (2020, methodology): “• AdaptSegNet [22] considers semantic segmentations as structured outputs that contain spatial similarities between source and target domains.”
Source-Free Domain Adaptation for Semantic Segmentation (2021, methodology): “For the former setting, we follow previous work [55, 41] by considering Cityscapes [8] as the target domain, and GTA5 [36] or SYNTHIA [37] as the source domain.”
Meta-Sim: Learning to Generate Synthetic Datasets (2019, background): “Addressing the domain gap issue has led to a plethora of work on synthetic-to-real domain adaptation [17, 27, 54, 10, 43, 34, 45].”
OpenEarthMap: A Benchmark Dataset for Global High-Resolution Land Cover Mapping (2022, methodology): “AdaptSeg [49] uses adversarial learning in the output space and a multi-level adversarial network to effectively perform output space domain adaptation at different feature levels.”
Adversarial Learning and Self-Teaching Techniques for Domain Adaptation in Semantic Segmentation (2019, methodology): “o the previously computed properties. Many other works addressed the domain adaptation problem with various techniques, including GANs [29], [45], cycle consistency [11], [46], output space alignment [47], [48], distillation loss [17], [49], class-balanced selftraining [50], conservative loss [51], ”
Unsupervised Domain Adaptation for Semantic Segmentation of Urban Scenes (2019, methodology): “To measure the performance, we compared the predictions on the Cityscapes validation set with the ground truth labels and computed the mean Intersection over Union (mIoU) as done by most competing approaches [12, 4, 35].”