DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effort

By Yuxuan Zhang, Huan Ling, Jun Gao, K. Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, Sanja Fidler. Computer Vision and Pattern Recognition, 2021

398 citations recorded by the source corpus.

We introduce DatasetGAN: an automatic procedure to generate massive datasets of high-quality semantically segmented images requiring minimal human effort. Current deep networks are extremely data-hungry, benefiting from training on large-scale datasets, which are time consuming to annotate. Our method relies on the power of recent GANs to generate realistic images. We show how the GAN latent code can be decoded to produce a semantic segmentation of the image. Training the decoder only needs a few labeled examples to generalize to the rest of the latent space, resulting in an infinite annotated dataset generator! These generated datasets can then be used for training any computer vision architecture just as real datasets are. As only a few images need to be manually segmented, it becomes possible to annotate images in extreme detail and generate datasets with rich object and part segmentations. To showcase the power of our approach, we generated datasets for 7 image segmentation tasks which include pixel-level labels for 34 human face parts, and 32 car parts. Our approach outperforms all semi-supervised baselines significantly and is on par with fully supervised methods, which in some cases require as much as 100x more annotated data as our method.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/786cb74290340443e9d02ffd9669f5e2a18878b5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold (2023, result): “While some previous works also leverage the discriminative feature in semantic segmentation [Tritrong et al. 2021; Zhang et al. 2021], we are the first to connect the point-based editing problem to the intuition of discriminative GAN features and design a concrete method.”
Your Diffusion Model is Secretly a Zero-Shot Classifier (2023, neutral): “Recent works on generative modeling have also learned ef-ﬁcient representations for both global and dense prediction tasks like classiﬁcation [28, 33, 13, 8, 19] and segmentation [46, 82, 10, 3, 9].”
High-fidelity GAN Inversion with Padding Space (2022, background): “Existing studies on GAN interpretation have affirmed that, pre-trained GAN models own great potential in a range of downstream applications, such as object classification [9, 45], semantic segmentation [49], video generation [23], and image editing [4, 11, 13, 20, 28, 31, 38, 39, 42, 46, 50].”
Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP (2021, methodology): “For faces, we evaluate our model on CelebA-Mask8 dataset [11], which contains 8 part categories and compare our method to state-of-the-art semi-supervised methods [10, 20].”
When, Why, and Which Pretrained GANs Are Useful? (2022, background): “…semantic editing (Isola et al., 2017; Zhu et al., 2018; Shen et al., 2020; Voynov & Babenko, 2020), image processing (Pan et al., 2020; Ledig et al., 2017; Menon et al., 2020), video generation (Wang et al., 2018a), producing high-quality synthetics (Zhang et al., 2021; Voynov et al., 2020).”
EditGAN: High-Precision Semantic Image Editing (2021, methodology): “Finally, a concurrent paper [72] shares similarities with DatasetGAN [1], on which our method builds, and explores an editing approach related to our EditGAN as one of its applications.”
GAN-Supervised Dense Visual Alignment (2021, methodology): “We do not rely on hand-crafted pixel space augmentations [12, 35], human-labeled data [76, 86, 87] or post-processing of GANgenerated datasets using domain knowledge [10, 55, 79, 86].”
Emergent Correspondence from Image Diffusion (2023, background): “In particular, recent work has also found that the internal representation of Generative Adversarial Networks (GAN) [24] could be used for identifying visual correspondence [94, 63, 58] within certain image categories.”
Repurposing GANs for One-Shot Semantic Part Segmentation (2021, methodology): “DatasetGAN [54] uses a similar representation extraction method to train segmentation networks in order to create high-detailed segmentation datasets with 16-40 annotations per object class.”
Label-Efficient Semantic Segmentation with Diffusion Models (2021, methodology): “In particular, the method proposed in (Zhang et al., 2021) trains a pixel class prediction model on these representations and conﬁrms their label efﬁciency.”
GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Images (2022, methodology): “To obtain the 2D segmentation of each image, we use DatasetGAN [35] to predict the 2D silhouette.”
BigDatasetGAN: Synthesizing ImageNet with Pixel-wise Annotations (2022, methodology): “We extend DatasetGAN [78] to synthesize ImageNet images with pixel-wise labels.”
Mind the Gap: Domain Gap Control for Single Shot Domain Adaptation for Generative Adversarial Networks (2021, background): “…activation space of StyleGAN have demonstrated that the GAN can be exploited for downstream tasks like unsupervised and few-shot part segmentation (Zhang et al., 2021; Tritrong et al., 2021; Abdal et al., 2021a; Collins et al., 2020; Bielski & Favaro, 2019), extracting 3D models of the objects…”
Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization (2021, background): “In parallel work, [97] exploit GANs to synthesize large labeled data datasets using very few labeled examples.”
Is synthetic data from generative models ready for image recognition? (2022, background): “Zhang et al. (2021) leverage the latent code of StyleGAN (Karras et al., 2019) to produce labels for object part segmentation.”
Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers (2021, methodology): “We construct three few-shot segmentation datasets by selecting training and validation images from CelebA-Faces, ADE-Cars, and LSUN-Cats as in Zhang et al. (2021b). The model is trained using cross-entropy loss.”