FCNs in the Wild: Pixel-level Adversarial and Constraint-based Adaptation

By Judy Hoffman, Dequan Wang, Fisher Yu, Trevor Darrell. 2016

830 citations recorded by the source corpus.

Fully convolutional models for dense prediction have proven successful for a wide range of visual tasks. Such models perform well in a supervised setting, but performance can be surprisingly poor under domain shifts that appear mild to a human observer. For example, training on one city and testing on another in a different geographic region and/or weather condition may result in significantly degraded performance due to pixel-level distribution shift. In this paper, we introduce the first domain adaptive semantic segmentation method, proposing an unsupervised adversarial approach to pixel prediction problems. Our method consists of both global and category specific adaptation techniques. Global domain alignment is performed using a novel semantic segmentation network with fully convolutional domain adversarial learning. This initially adapted space then enables category specific adaptation through a generalization of constrained weak learning, with explicit transfer of the spatial layout from the source to the target domains. Our approach outperforms baselines across different settings on multiple large-scale datasets, including adapting across various real city environments, different synthetic sub-domains, from simulated to real environments, and on a novel large-scale dash-cam dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Domain Adaptation for Traffic Density Estimation (2021, neutral): “The first to apply such a technique is (Hoffman et al., 2016).”
Stacked Generative Adversarial Networks (2016, background): “The idea of training a discriminator in some representation space has been explored in the context of domain adaptation [12, 2, 24], and regularization of auto-encoders [38].”
Recognition in Terra Incognita (2018, neutral): “[45,46,47] investigate methods of domain adaptation for semantic segmentation, focusing mainly on cars and pedestrians and either adapting from synthetic to real data, from urban to suburban scenes, or from PASCAL to a camera on-board a car.”
Unsupervised Model Adaptation for Continual Semantic Segmentation (2020, methodology): “We have also included methods that are based on direct distributional matching which are more similar to MAS3: FCNs in the Wild (Hoffman et al. 2016), CDA (Zhang, David, and Gong 2017), DCAN (Wu et al. 2018), SWD (Lee et al. 2019), Cross-City (Chen et al. 2017b).”
CyCADA: Cycle-Consistent Adversarial Domain Adaptation (2017, background): “More recently, a convolutional domain adversarial based approached was proposed for more general drive cam scenes and for adaptation from simulated to real environments (Hoffman et al., 2016).”
Contrastive Adaptation Network for Unsupervised Domain Adaptation (2019, methodology): “In the absence of labeled data from the target domain, Unsupervised Domain Adaptation (UDA) methods have emerged to mitigate the domain shift in data distributions [2, 1, 5, 32, 25, 17, 3, 16].”
Generalizing to Unseen Domains via Adversarial Data Augmentation (2018, methodology): “We evaluate our method for both classification and semantic segmentation settings, following the evaluation scenarios of domain adaptation techniques [9, 39, 14], though in our case the target domains are unknown at training time.”
No More Discrimination: Cross City Adaptation of Road Scene Segmenters (2017, methodology): “[11] extend such frameworks for semantic segmentation.”
Domain Adaptive Faster R-CNN for Object Detection in the Wild (2018, background): “Recently there are some works concerning tasks such as semantic segmentation [4, 27, 61], fine-grained recognition [16] etc.”
ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation (2018, methodology): “Main approaches for UDA include discrepancy minimization between source and target feature distributions [10, 24, 15, 25, 42], self-training with pseudo-labels [51] and generative approaches [14, 34, 43].”
FDA: Fourier Domain Adaptation for Semantic Segmentation (2020, background): “Alignment in high-level feature space [27, 16, 36, 38, 31] can be counter-productive for semantic segmentation, unlike image-level classification, [20, 28, 37], due to the complex representations and the difficulty in stabilizing adversarial training.”
Adversarial Dropout Regularization (2017, methodology): “DANN and FCN Wild denote methods proposed by (Ganin & Lempitsky (2014)) and (Hoffman et al. (2016) respectively.”
Uncertainty-aware multi-view co-training for semi-supervised medical image segmentation and domain adaptation (2020, background): “Recent works [14,17,18,24,25,55,61,72,73] utilizes adversarial training and self-training to adapt feature training between source domain and target domain.”
Boundary-weighted Domain Adaptive Neural Network for Prostate MR Image Segmentation (2019, background): “Hoffman et al. [41] presented an unsupervised domain adaptation framework with fully convolutional networks for semantic segmentation.”
VisDA: The Visual Domain Adaptation Challenge (2017, methodology): “The adaptation method in [16] improves mIoU to 25.5.”
Adaptation Across Extreme Variations using Unlabeled Domain Bridges (2019, methodology): “Deep domain adaptation that realizes UDA in deep learning framework has been successful in several vision tasks [54, 15, 8, 26, 24, 23, 51].”
Exploring Categorical Regularization for Domain Adaptive Object Detection (2020, background): “In the literature, adversarial training has been well-studied for domain adaptive image classification [5, 6, 22, 33], semantic segmentation [13, 30, 32] and object detection [1, 28, 41, 12].”
ROAD: Reality Oriented Adaptation for Semantic Segmentation of Urban Scenes (2017, methodology): “Among those, the case of 1×1 is the ordinary domain adaptation method without considering the spatial information, which is also equivalent to the global alignment method in [17].”