Context-aware Feature Generation For Zero-shot Semantic Segmentation

By Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, Liqing Zhang. 2020

164 citations recorded by the source corpus.

Existing semantic segmentation models heavily rely on dense pixel-wise annotations. To reduce the annotation pressure, we focus on a challenging task named zero-shot semantic segmentation, which aims to segment unseen objects with zero annotations. This task can be accomplished by transferring knowledge across categories via semantic word embeddings. In this paper, we propose a novel context-aware feature generation method for zero-shot segmentation named CaGNet. In particular, with the observation that a pixel-wise feature highly depends on its contextual information, we insert a contextual module in a segmentation network to capture the pixel-wise contextual information, which guides the process of generating more diverse and context-aware features from semantic word embeddings. Our method achieves state-of-the-art results on three benchmark datasets for zero-shot segmentation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chen Change Loy
2. Zheng Zhang
3. Han Hu
4. Luc Van Gool
5. Dacheng Tao
6. Jiashi Feng
7. Yue Cao
8. Jianfeng Gao

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Ming-Hsuan Yang
4. Chen Change Loy
5. Chunhua Shen
6. Xiaogang Wang
7. Alan L. Yuille
8. Jitendra Malik

In Meta & Continual Learning

1. René Ranftl
2. Jun Liu
3. Bo Dai
4. Vladlen Koltun
5. Chen Change Loy
6. Dengxin Dai
7. Jian Ding
8. Ping Hu

Citations in the corpus

• Language-driven Semantic Segmentation (2022, methodology): “Bucher et al. (2019) and Gu et al. (2020) propose to leverage word embeddings together with a generative model to generate visual features of unseen categories, while Xian et al. (2019) propose to project visual features into a simple word embedding space and to correlate the resulting embeddings…”
• ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, methodology): “In such “transductive” situation, both the ground truth of seen classes, as well as pseudo labels of unseen classes, will be utilized to supervise the target model for self-training [17,49,56] which can achieve better performance.”
• DenseCLIP: Extract Free Dense Labels from CLIP (2021, methodology): “Other studies like [26], [34], and [20], improve the generative ZS3Net in terms of uncertainty, structural consistency, and context, respectively, while STRICT [41] boosts the SPNet through self-training.”
• Extract Free Dense Labels from CLIP (2021, methodology): “We compare MaskCLIP+ with SOTA methods including SPNet [50], ZS3Net [1], CaGNet [20], and STRICT [41].”
• Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields (2023, methodology): “In our experiments, we employ SAM [17] and LSeg [20], facilitating promptable, promptless (zero-shot [3] [12]) and language-driven computer vision tasks in a 3D context.”
• Conterfactual Generative Zero-Shot Semantic Segmentation (2021, methodology): “Most of the new works follow these two works and import new knowledge resources like context [14, 24] and relationships between classes [26] to help generate more fitting fake features.”
• A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model (2021, methodology): “CaGNet [20,21] introduce a contextual module for better feature generation.”
• Decoupling Zero-Shot Semantic Segmentation (2021, methodology): “Since most of the previous works [7,20,42] focus on the GZS3 setting, we evaluate our method on the GZS3 setting in the main paper.”