Decoupling Zero-Shot Semantic Segmentation

By Jian Ding, Nan Xue, Guisong Xia, Dengxin Dai. 2021

305 citations recorded by the source corpus.

Zero-shot semantic segmentation (ZS3) aims to segment the novel categories that have not been seen in the training. Existing works formulate ZS3 as a pixel-level zeroshot classification problem, and transfer semantic knowledge from seen classes to unseen ones with the help of language models pre-trained only with texts. While simple, the pixel-level ZS3 formulation shows the limited capability to integrate vision-language models that are often pre-trained with image-text pairs and currently demonstrate great potential for vision tasks. Inspired by the observation that humans often perform segment-level semantic labeling, we propose to decouple the ZS3 into two sub-tasks: 1) a classagnostic grouping task to group the pixels into segments. 2) a zero-shot classification task on segments. The former task does not involve category information and can be directly transferred to group pixels for unseen classes. The latter task performs at segment-level and provides a natural way to leverage large-scale vision-language models pre-trained with image-text pairs (e.g. CLIP) for ZS3. Based on the decoupling formulation, we propose a simple and effective zero-shot semantic segmentation model, called ZegFormer, which outperforms the previous methods on ZS3 standard benchmarks by large margins, e.g., 22 points on the PAS-CAL VOC and 3 points on the COCO-Stuff in terms of mIoU for unseen classes. Code will be released at https://github.com/dingjiansw101/ZegFormer.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

Generalized Decoding for Pixel, Image, and Language (2022, neutral): “8 ⋆ ⋆ - ⋆ ⋆ - - ⋆ ZegFormer (B) [15] ✓ ✓ ✗ ✗ ✓ ✓ ✗ - - ⋆ 8.”
ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, methodology): “The unseen classes of each dataset are shown in the Appendix according to previous works [12, 17, 49, 56].”
MaPLe: Multi-modal Prompt Learning (2022, methodology): “Many works have demonstrated better performance on downstream tasks by using tailored methods to adapt V-L models for few-shot image-recognition [9, 19, 46], object detection [8,10,27,34,44,50], and segmentation [5,22,26,33].”
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, background): “As large vision-language models gained popularity for image classification, numerous new approaches [11, 20, 23, 26, 32, 40, 43, 45, 48, 56, 71, 72, 79, 80] have emerged to tackle open-vocabulary or zero-shot image semantic segmentation.”
ConceptFusion: Open-set Multimodal 3D Mapping (2023, methodology): “Image-language aligned models like CLIP [6] and ALIGN [9] encode text and images into a shared concept space, and have driven progress in several open-set tasks [11, 12, 13, 14, 15, 16, 17, 18].”
Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models (2023, background): “We find that our proposed approach of pooling the masked features is more efficient and yet as effective as the alternative approach proposed in [15,25], which crops each of the N predicted masks’ regions from the original image and encodes them separately.”
Self-regulating Prompts: Foundational Model Adaptation without Forgetting (2023, background): “VL models possess a strong understanding of open-vocabulary concepts, making them suitable for various downstream vision and vision-language applications [12, 56, 38, 30, 60, 13, 32, 53, 26, 36, 8].”
Side Adapter Network for Open-Vocabulary Semantic Segmentation (2023, background): “Early attempts [9, 33] adopt a two-stage training framework.”
Vision-Language Models for Vision Tasks: A Survey (2023, background): “The second line explores VLMs with knowledge distillation [35], [36], [37], e.g. , several studies [35], [36], [37] explore how to distill knowledge from VLMs to downstream tasks, aiming for better performance in object detection, semantic segmentation, etc.”
Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP (2022, methodology): “5 ZegFormer [11] R-50 COCO-Stuff-156 - 16.”