Repurposing GANs for One-Shot Semantic Part Segmentation

By Pitchaporn Rewatbowornwong, Nontawat Tritrong, Supasorn Suwajanakorn. 2021

127 citations recorded by the source corpus.

While GANs have shown success in realistic image generation, the idea of using GANs for other tasks unrelated to synthesis is underexplored. Do GANs learn meaningful structural parts of objects during their attempt to reproduce those objects? And can image synthesis serve as an “upstream” representation learning task? In this work, we test these hypotheses and propose a simple and effective approach based on GANs for fundamental vision tasks: semantic part segmentation and landmark detection. With our approach, these tasks only require as few as one labeled example along with an unlabeled dataset, rather than thousands of examples. Our key idea is to leverage a trained GAN to extract a pixel-wise representation from the input image and use it as feature vectors for a segmentation network. Our experiments demonstrate that this GAN-derived representation is “readily discriminative” and produces surprisingly good results that are comparable to those from supervised baselines trained with significantly more labels. We believe this novel repurposing of GANs underlies a new class of unsupervised representation learning, which can generalize to many other tasks. More results are available at https://RepurposeGANs.github.io/.

Subfields: Computer Vision and Pattern Recognition, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Generative Models

Citations in the corpus

InstructPix2Pix: Learning to Follow Image Editing Instructions (2022, background): “As generative models continue to improve, there is growing interest in their use as a source of cheap and plentiful training data for downstream tasks [33, 47, 50, 58, 65, 66].”
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models (2023, methodology): “There exist prior works, which are similar in spirit to ours in their use of image generative models, including GANs [3,19,35,36,98] and diffusion models [14, 17, 31, 34, 60, 69–73, 77] to perform semantic segmentation [2,21,41,54,76,92].”
Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold (2023, result): “While some previous works also leverage the discriminative feature in semantic segmentation [Tritrong et al. 2021; Zhang et al. 2021], we are the first to connect the point-based editing problem to the intuition of discriminative GAN features and design a concrete method.”
Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks (2021, background): “…StyleGAN have demonstrated that the GAN can be exploited for downstream tasks like unsupervised and few-shot part segmentation (Zhang et al., 2021; Tritrong et al., 2021; Abdal et al., 2021a; Collins et al., 2020; Bielski & Favaro, 2019), extracting 3D models of the objects (Pan et al., 2021;…”
BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations (2022, unknown): “Works like [23, 47, 71] use an encoder to embed real images into a GAN’s latent space and train a segmentation branch on top of the GAN’s features to produce pixel-wise labeling of real images.”
GAN-Supervised Dense Visual Alignment (2021, methodology): “We do not rely on hand-crafted pixel space augmentations [12, 35], human-labeled data [76, 86, 87] or post-processing of GANgenerated datasets using domain knowledge [10, 55, 79, 86].”
Ensembling with Deep Generative Views (2021, methodology): “Augmentation using GANs can be applied at training time for robustness or test-time for ensembling; while we primarily focus on the latter, concurrent works [35, 50] investigate the benefits of the former approach while [56] investigates intermediate GAN representations for few-shot segmentation.”
Label-Efficient Semantic Segmentation with Diffusion Models (2021, background): “The second line of works (Zhang et al., 2021; Tritrong et al., 2021; Xu, 2021; Galeev et al., 2020) is more relevant to our study since they are based on the intermediate representations obtained in GANs.”
Synthetic Data from Diffusion Models Improves ImageNet Classification (2023, methodology): “Examples include tasks like semantic image segmentation [3, 10, 36, 37, 54, 66], optical flow estimation [14, 32, 63], human motion understanding [19, 29, 38, 67], and other dense prediction tasks [3, 70].”