Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization

By Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, Sanja Fidler. 2021

232 citations recorded by the source corpus.

Training deep networks with limited labeled data while achieving a strong generalization ability is key in the quest to reduce human annotation efforts. This is the goal of semi-supervised learning, which exploits more widely available unlabeled data to complement small labeled data sets. In this paper, we propose a novel framework for discriminative pixel-level tasks using a generative model of both images and labels. Concretely, we learn a generative adversarial network that captures the joint image-label distribution and is trained efficiently using a large set of un-labeled images supplemented with only few labeled ones. We build our architecture on top of StyleGAN2 [45], augmented with a label synthesis branch. Image labeling at test time is achieved by first embedding the target image into the joint latent space via an encoder network and test-time optimization, and then generating the label from the inferred embedding. We evaluate our approach in two important domains: medical image segmentation and part-based face segmentation. We demonstrate strong in-domain performance compared to several baselines, and are the first to showcase extreme out-of-domain generalization, such as transferring from CT to MRI in medical imaging, and photographs of real faces to paintings, sculptures, and even cartoons and animal faces. Project Page: https://nv-tlabs.github.io/semanticGAN/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

Score-based Generative Modeling in Latent Space (2021, background): “Furthermore, like other deep generative models, it can potentially be used also for non-generative tasks such as semi-supervised and representation learning [101, 102, 103].”
Generalizing to Unseen Domains: A Survey on Domain Generalization (2021, unknown): “Similar spirits also include [129], [130].”
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (2022, methodology): “In particular, we follow SemanticGAN [21] and use StyleGAN2 [19] backbone.”
EditGAN: High-Precision Semantic Image Editing (2021, methodology): “To train this encoder we mainly follow SemanticGAN [2], which builds on [62], with further improvements.”
Label-Efficient Semantic Segmentation with Diffusion Models (2021, background): “While previous works highlighted the practical advantages of these representations, such as out-of-distribution robustness (Li et al., 2021a), generative models as representation learners receive less attention compared to alternative unsupervised methods, e.g., based on contrastive learning (Chen…”
Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP (2021, methodology): “For faces, we evaluate our model on CelebA-Mask8 dataset [11], which contains 8 part categories and compare our method to state-of-the-art semi-supervised methods [10, 20].”
BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations (2022, methodology): “Works like [23, 47, 71] use an encoder to embed real images into a GAN’s latent space and train a segmentation branch on top of the GAN’s features to produce pixel-wise labeling of real images.”
Your Diffusion Model is Secretly a Zero-Shot Classifier (2023, unknown): “Recent works on generative modeling have also learned ef-ﬁcient representations for both global and dense prediction tasks like classiﬁcation [28, 33, 13, 8, 19] and segmentation [46, 82, 10, 3, 9].”
Synthetic Data from Diffusion Models Improves ImageNet Classification (2023, methodology): “Examples include tasks like semantic image segmentation [3, 10, 36, 37, 54, 66], optical flow estimation [14, 32, 63], human motion understanding [19, 29, 38, 67], and other dense prediction tasks [3, 70].”
DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort (2021, background): “In parallel work [31], the authors explore an alternative direction in which the GAN, equipped with a segmentation branch, is also used as a semantic decoder at test time.”