Learning from Synthetic Data: Addressing Domain Shift for Semantic Segmentation

By S. Sankaranarayanan, Yogesh Kumar Balaji, Arpit Jain, Ser-Nam Lim, Ramalingam Chellappa. 2017

527 citations recorded by the source corpus.

Visual Domain Adaptation is a problem of immense importance in computer vision. Previous approaches showcase the inability of even deep neural networks to learn informative representations across domain shift. This problem is more severe for tasks where acquiring hand labeled data is extremely hard and tedious. In this work, we focus on adapting the representations learned by segmentation networks across synthetic and real domains. Contrary to previous approaches that use a simple adversarial objective or superpixel information to aid the process, we propose an approach based on Generative Adversarial Networks (GANs) that brings the embeddings closer in the learned feature space. To showcase the generality and scalability of our approach, we show that we can achieve state of the art results on two challenging scenarios of synthetic to real domain adaptation. Additional exploratory experiments show that our approach: (1) generalizes to unseen domains and (2) results in improved alignment of source and target distributions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Unsupervised Model Adaptation for Continual Semantic Segmentation (2020, methodology): “A large group of the existing UDA algorithms for image segmentation use adversarial learning for extracting domaininvariant features (Luc et al. 2016; Bousmalis et al. 2017; Hoffman et al. 2018b; Murez et al. 2018; Saito et al. 2018; Sankaranarayanan et al. 2018; Dhouib, Redko, and Lartizien 2020).”
ADVENT: Adversarial Entropy Minimization for Domain Adaptation in Semantic Segmentation (2018, methodology): “Other approaches include self-training [51] to provide pseudo labels or generative networks to produce target data [14, 34, 43].”
CrDoCo: Pixel-Level Domain Transfer With Cross-Domain Consistency (2019, methodology): “Results on both feature backbones show that our method performs favorably against the state-of-the-art methods, outperforming the previous best competitors by 4.9% in mean IoU [17] when using the DRN-26 [46] and 1.0% in mean IoU [35] when using FCN8s-VGG16 [25].”
What Can Be Transferred: Unsupervised Domain Adaptation for Endoscopic Lesions Segmentation (2020, unknown): “[22] first utilize adversarial network [19] to achieve domain adaptation for semantic segmentation task, diverse variants based on adversarial strategy [11,23,34,35, 37, 41] are proposed to address the domain shift challenge.”
FDA: Fourier Domain Adaptation for Semantic Segmentation (2020, background): “6 LSD[37] 88.”
ACDC: The Adverse Conditions Dataset with Correspondences for Semantic Driving Scene Understanding (2021, background): “A major class of UDA works employs adversarial domain adaptation to implicitly align the source and target domains at the level of pixels and/or features [7, 15, 26, 41, 43, 44, 46, 48, 60].”
Attending to Discriminative Certainty for Domain Adaptation (2019, methodology): “Image generation methods are used to adapt the source and target domain [13, 38, 30].”
AuxNet: Auxiliary tasks enhanced Semantic Segmentation for Automated Driving (2019, methodology): “One way to overcome this is via using synthetic datasets and domain adaptation techniques (Sankaranarayanan et al., 2018), another way is to use”
Fake it till you make it: face analysis in the wild using synthetic data alone (2021, background): “Instead of adapting data, it is possible to learn features that are resistant to the differences between domains [13, 57].”
Survey on semantic segmentation using deep learning techniques (2019, methodology): “The authors in [222] proposed a domain Shift approach based on Generative Adversarial Network (GAN), which transfers the information of the target distribution to the learned embedding using a generator-discriminator pair.”
MetaReg: Towards Domain Generalization using Meta-Regularization (2018, methodology): “Other notable works include the use of Maximum Mean Discrepancy (MMD) [21], Generative Adversarial Networks (GAN) [25][26], co-training [4], etc.”
Semi-Supervised and Unsupervised Deep Visual Learning: A Survey (2022, background): “These models typically learn a real-to-real [212], [213], [214] or synthetic-to-real [210], [211] mapping, rendering the style of the labeled source to the unlabeled target domain, thus offering synthetic training data with pseudo labels.”
Synergistic Image and Feature Adaptation: Towards Cross-Modality Domain Adaptation for Medical Image Segmentation (2019, background): “Sankaranarayanan et al. reconstructs the features into images and put a discriminator in the reconstructed image space (Sankaranarayanan et al. 2018).”
Sliced Wasserstein Discrepancy for Unsupervised Domain Adaptation (2019, background): “Other hybrid approaches have also been proposed in [60, 25].”
Detection and Retrieval of Out-of-Distribution Objects in Semantic Segmentation (2020, background): “Publications in this direction include the rendering of highly realistic images [35] or using generative adversarial networks (GANs) for generation and augmentation of already collected data [30].”
Deep Visual Domain Adaptation: A Survey (2018, methodology): “ce network, and a spatial-aware adaptation module leverages the intrinsic spatial structure to reduce the domain divergence. Rather than operating a simple adversarial objective on the feature space, [100] used a GAN Manuscript accepted by Neurocomputing 2018 16 to address domain shift in which a ge”