Zero-Shot Semantic Segmentation

By Max Bucher, Tuan-Hung Vu, Matthieu Cord, Patrick Pérez. 2019

406 citations recorded by the source corpus.

Semantic segmentation models are limited in their ability to scale to large numbers of object classes. In this paper, we introduce the new task of zero-shot semantic segmentation: learning pixel-wise classifiers for never-seen object categories with zero training examples. To this end, we present a novel architecture, ZS3Net, combining a deep visual segmentation model with an approach to generate visual representations from semantic word embeddings. By this way, ZS3Net addresses pixel classification tasks where both seen and unseen categories are faced at test time (so called "generalized" zero-shot classification). Performance is further improved by a self-training step that relies on automatic pseudo-labeling of pixels from unseen classes. On the two standard segmentation datasets, Pascal-VOC and Pascal-Context, we propose zero-shot benchmarks and set competitive baselines. For complex scenes as ones in the Pascal-Context dataset, we extend our approach by using a graph-context encoding to fully leverage spatial context priors coming from class-wise segmentation maps.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Language-driven Semantic Segmentation (2022, methodology): “Bucher et al. (2019) and Gu et al. (2020) propose to leverage word embeddings together with a generative model to generate visual features of unseen categories, while Xian et al. (2019) propose to project visual features into a simple word embedding space and to correlate the resulting embeddings…”
PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, background): “Further works should investigate how to exploit the already present future information with Zeroshot [42, 41] as already done in semantic segmentation [36, 5] and explored for continual classification [72, 19].”
GroupViT: Semantic Segmentation Emerges from Text Supervision (2022, background): “Going further, zero-shot approaches [6, 38] are proposed to learn segmentation models for unseen categories without using pixel-wise labels for them.”
Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP (2022, methodology): “First of all, compared with perpixel approaches (SPNet [37], ZS3Net [4], LSeg [23] and LSeg+ [16]), proposal-based approaches (OpenSeg [16], SimBaseline [40] and ZegFormer [11]) show better performance.”
A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model (2021, methodology): “Different from [5,32,20], SPNet [48] attempt to mapping vision feature to the semantic space via word embedding.”
ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, methodology): “This gives rise to a surging interest in low-supervision-based semantic segmentation approaches, including semi-supervised [7], weaklysupervised [48], few-shot [46], and zero-shot semantic segmentation [3, 34, 44].”
Scaling Open-Vocabulary Image Segmentation with Image-Level Labels (2021, methodology): “3 ZS3Net [6] ResNet101 7 3(seen classes) - - 19.”
Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (2023, methodology): “In our experiments, we employ SAM [17] and LSeg [20], facilitating promptable, promptless (zero-shot [3] [12]) and language-driven computer vision tasks in a 3D context.”
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation (2022, background): “Performing L-ZSON in any environment with unstructured language input is challenging; however, recent advances in open-vocabulary models for image classification [34, 55, 57], object detection [3, 20, 26, 35, 41, 45, 47, 58, 82], and semantic segmentation [2, 5, 14, 32, 35, 36, 81] present a promisi”
Normalization Matters in Zero-Shot Learning (2020, background): “Zero-shot learning (ZSL) has a long-standing history [41] and nowadays people employ zero-shot models not only for image classification, but also for object detection, text classification, named entity recognition, semantic segmentation and many other tasks [52, 4, 5].”
Consistent Structural Relation Learning for Zero-Shot Segmentation (2020, methodology): “Most recent works [1, 26] further extend the zero-shot learning to the semantic segmentation task.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, background): “Zero-shot semantic segmentation is a challenging task [24, 2, 10] where a model has to predict semantic labels of pixels in images without a-priori information of the categories.”
Uncertainty-Aware Learning for Zero-Shot Semantic Segmentation (2020, methodology): “To achieve effective zero-shot semantic segmentation (ZSS), existing efforts have been made [2, 28, 31, 44, 55, 62] by treating each pixel as an independent classiﬁcation problem.”
A Review of Generalized Zero-Shot Learning Methods (2020, methodology): “In addition, GZSL has been applied to segment both seen and unseen categories [192], [193], retrieve images from large scale data”
DenseCLIP: Extract Free Dense Labels from CLIP (2021, methodology): “ZS3Net [1] proposes to generate fake pixel-level features from semantic features for the unseen.”
Extract Free Dense Labels from CLIP (2021, methodology): “Other studies like [26], [34], and [20], improve the generative ZS3Net in terms of uncertainty, structural consistency, and context, respectively, while STRICT [41] boosts the SPNet through self-training.”