Segmentation in Style: Unsupervised Semantic Image Segmentation with Stylegan and CLIP

By Daniil Pakhomov, Sanchit Hira, Narayani Wagle, Kemar E. Green, Nassir Navab. 2021

36 citations recorded by the source corpus.

We introduce a method that allows to automatically segment images into semantically meaningful regions without human supervision. Derived regions are consistent across different images and coincide with human-defined semantic classes on some datasets. In cases where semantic regions might be hard for human to define and consistently label, our method is still able to find meaningful and consistent semantic classes. In our work, we use pretrained StyleGAN2 generative model: clustering in the feature space of the generative model allows to discover semantic classes. Once classes are discovered, a synthetic dataset with generated images and corresponding segmentation masks can be created. After that a segmentation model is trained on the synthetic dataset and is able to generalize to real images. Additionally, by using CLIP we are able to use prompts defined in a natural language to discover some desired semantic classes. We test our method on publicly available datasets and show state-of-the-art results.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Chunyuan Li
2. Jiuxiang Gu
3. Changyou Chen
4. Tong Yu
5. Tong Sun
6. Aditya Sanghi
7. Yufan Zhou
8. Joseph G. Lambourne

Citations in the corpus

• Towards Language-Free Training for Text-to-Image Generation (2021, background): “The idea of multimodal feature space also inspires some recent works on generative models [9, 10, 33, 35].”
• ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, methodology): “Recently, the pretrained vision-language model CLIP [36] has been adopted into various dense prediction tasks, such as referring segmentation [43], semantic segmentation [33], and detection [14].”
• CLIP-Forge: Towards Zero-Shot Text-to-Shape Generation (2021, methodology): “Recently, pre-trained CLIP has been used for several zero-shot downstream applications [16, 18, 26, 41,45,54].”
• Repurposing GANs for One-Shot Semantic Part Segmentation (2021, unknown): “Segmentation in style [57] clustersGAN’s features fromdifferent”