Zero-Shot Semantic Segmentation via Variational Mapping

By Naoki Kato, Toshihiko Yamasaki, Kiyoharu Aizawa. 2019

55 citations recorded by the source corpus.

We have witnessed the explosive success of deep neural networks (DNNs). However, DNNs typically assume a large amount of training data, and this is not always available in practical scenarios. In this paper, we present zero-shot semantic segmentation, where a model that has never seen the target class during training. For this purpose, we propose variational mapping, which facilitates effective learning by mapping the class label embedding vectors from the semantic space to the visual space. Experimental results using Pascal VOC 2012 show that our proposed method can achieve a mean intersection over union (mIoU) of 42.2, and we believe that this can serve as a baseline for similar research in the future.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, background): “Further works should investigate how to exploit the already present future information with Zeroshot [42, 41] as already done in semantic segmentation [36, 5] and explored for continual classification [72, 19].”
A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model (2021, methodology): “In [26], variational mapping is used to learn semantic features.”
Sketch-a-Segmenter: Sketch-Based Photo Segmenter Generation (2020, methodology): “Zero-shot learning has also been extended to the task of photo segmentation [10]–[12].”
CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, methodology): “Zero-shot techniques have become increasingly popular in image classification [53, 33, 51], object detection [4, 54, 19, 38, 77, 42, 26], and semantic segmentation [6, 35, 32, 3, 14, 76].”
DenseCLIP: Extract Free Dense Labels from CLIP (2021, background): “[32] fuse semantic features into visual features as guidance.”
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation (2022, methodology): “Performing L-ZSON in any environment with unstructured language input is challenging; however, recent advances in open-vocabulary models for image classification [34, 55, 57], object detection [3, 20, 26, 35, 41, 45, 47, 58, 82], and semantic segmentation [2, 5, 14, 32, 35, 36, 81] present a promisi”
Extract Free Dense Labels from CLIP (2021, background): “[32] fuse semantic features into visual features as guidance.”
Uncertainty-Aware Learning for Zero-Shot Semantic Segmentation (2020, methodology): “To achieve effective zero-shot semantic segmentation (ZSS), existing efforts have been made [2, 28, 31, 44, 55, 62] by treating each pixel as an independent classiﬁcation problem.”