Semantic Projection Network for Zero- and Few-Label Semantic Segmentation

By Yongqin Xian, Subhabrata Choudhury, Yang He, Bernt Schiele, Zeynep Akata. 2019

294 citations recorded by the source corpus.

Semantic segmentation is one of the most fundamental problems in computer vision and pixel-level labelling in this context is particularly expensive. Hence, there have been several attempts to reduce the annotation effort such as learning from image level labels and bounding box annotations. In this paper we take this one step further and focus on the challenging task of zero- and few-shot learning of semantic segmentation. We define this task as image segmentation by assigning a label to every pixel even though either no labeled sample of that class was present during training, i.e. zero-label semantic segmentation, or only a few labeled samples were present, i.e. few-label semantic segmentation.Our goal is to transfer the knowledge from previously seen classes to novel classes. Our proposed semantic projection network (SPNet) achieves this goal by incorporating a class-level semantic information into any network designed for semantic segmentation, in an end-to-end manner. We also propose a benchmark for this task on the challenging COCO-Stuff and PASCAL VOC12 datasets. Our model is effective in segmenting novel classes, i.e. alleviating expensive dense annotations, but also in adapting to novel classes without forgetting its prior knowledge, i.e. generalized zero- and few-label semantic segmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Meta & Continual Learning

Citations in the corpus

Language-driven Semantic Segmentation (2022, methodology): “…with the CLIP image encoder with the same setup and hyperparameters, but observed worse performance than using the ViT initialization. to the competitive zero-shot baseline ZS3Net (Bucher et al., 2019), which adopts the DeepLabv3+ framework and to Xian et al. (2019) which leverages DeepLabv2.”
A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model (2021, methodology): “For zero-shot setting , we use harmonic mean IoU (hIoU) among the seen classes and unseen classes as major metric by following previous works [48,40] (see Appendix for detail definition).”
Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP (2022, result): “First of all, compared with perpixel approaches (SPNet [37], ZS3Net [4], LSeg [23] and LSeg+ [16]), proposal-based approaches (OpenSeg [16], SimBaseline [40] and ZegFormer [11]) show better performance.”
Decoupling Zero-Shot Semantic Segmentation (2021, methodology): “(1) They usually transfer knowledge from seen to unseen classes by language models [7,27,54] pre-trained only with texts, which limit their performance on vision tasks.”
Zero-Shot Semantic Segmentation via Variational Mapping (2019, methodology): “We plan to investigate similarities and differences between our method and [32] in future work.”
Image Segmentation Using Text and Image Prompts (2021, background): “Others add semantic class information into dense prediction models [27].”
Uncertainty-Aware Learning for Zero-Shot Semantic Segmentation (2020, methodology): “To achieve effective zero-shot semantic segmentation (ZSS), existing efforts have been made [2, 28, 31, 44, 55, 62] by treating each pixel as an independent classiﬁcation problem.”
Side Adapter Network for Open-Vocabulary Semantic Segmentation (2023, background): “Open-vocabulary Semantic Segmentation Earlier work [3, 31, 40] for open-vocabulary semantic segmentation focus on learning a joint embedding space between image pixels and class name/description.”
ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, methodology): “Our proposed method follows the generalized zero-shot semantic segmentation (GZLSS) [44], which requires to segment both seen classes Cs and unseen classes Cu after only training on a dataset with pixel-annotations of seen part.”
Scaling Open-Vocabulary Image Segmentation with Image-Level Labels (2021, methodology): “3 SPNet [49] ResNet101 7 3(seen classes) - - 24.”
Extract Free Dense Labels from CLIP (2021, methodology): “To relax MaskCLIP from the architectural constraint and to incorporate more advanced architectures such as PSPNet [55] and DeepLab [5,6], we notice that instead of deploying MaskCLIP at the inference time, we can deploy it at the training time, where it serves as a generalizable and robust annotator”
Consistent Structural Relation Learning for Zero-Shot Segmentation (2020, methodology): “Most recent works [1, 26] further extend the zero-shot learning to the semantic segmentation task.”
Open Vocabulary Semantic Segmentation with Patch Aligned Contrastive Learning (2022, result): “Note that the numbers shown for SPNet, ZS3Net and ViL-Seg are obtained from the ViL-Seg paper [32] and the numbers for all other baselines are obtained from their respective papers (cited in the table).”
Context-aware Feature Generation For Zero-shot Semantic Segmentation (2020, background): “SPNet reports results on large-scale COCO-stuff dataset[4], which makes their results more convincing.”
Open-Vocabulary Image Segmentation (2021, methodology): “These baselines have already surpassed the performances in existing zero-shot image segmentation meth-ods [4, 6, 19, 28, 47] on several datasets due to large-scale image-text pre-training.”
DenseCLIP: Extract Free Dense Labels from CLIP (2021, methodology): “SPNet [50] learns a projection from visual space to semantic space.”
A Few Guidelines for Incremental Few-Shot Segmentation (2020, methodology): “9 SPNet [42] 62.”