A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model

By Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, Xiang Bai. European Conference on Computer Vision, 2021

346 citations recorded by the source corpus.

Recently, open-vocabulary image classification by vision language pre-training has demonstrated incredible achievements, that the model can classify arbitrary categories without seeing additional annotated images of that category. However, it is still unclear how to make the open-vocabulary recognition work well on broader vision problems. This paper targets open-vocabulary semantic segmentation by building it on an off-the-shelf pre-trained vision-language model, i.e., CLIP. However, semantic segmentation and the CLIP model perform on different visual granularity, that semantic segmentation processes on pixels while CLIP performs on images. To remedy the discrepancy in processing granularity, we refuse the use of the prevalent one-stage FCN based framework, and advocate a two-stage semantic segmentation framework, with the first stage extracting generalizable mask proposals and the second stage leveraging an image based CLIP model to perform open-vocabulary classification on the masked image crops which are generated in the first stage. Our experimental results show that this two-stage framework can achieve superior performance than FCN when trained only on COCO Stuff dataset and evaluated on other datasets without fine-tuning. Moreover, this simple framework also surpasses previous state-of-the-arts of zero-shot semantic segmentation by a large margin: +29.5 hIoU on the Pascal VOC 2012 dataset, and +8.9 hIoU on the COCO Stuff dataset. With its simplicity and strong performance, we hope this framework to serve as a baseline to facilitate future research. The code are made publicly available at~\url{https://github.com/MendelXu/zsseg.baseline}.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/ae92c1fd30ba49212e16152646e9f7f1ae385b08

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chen Change Loy
2. Kai Chen
3. Ziwei Liu
4. Han Hu
5. Fangyun Wei
6. Zheng Zhang
7. Jiangmiao Pang
8. Wenwei Zhang

In Computer Vision and Pattern Recognition

1. Chen Change Loy
2. Luc Van Gool
3. Xiaogang Wang
4. Chunhua Shen
5. Dacheng Tao
6. Ming-Hsuan Yang
7. Alan L. Yuille
8. Xiang Bai

In Radiology, Nuclear Medicine and Imaging

1. Ziwei Liu
2. Xiang Bai
3. Kai Chen
4. Chen Change Loy
5. Han Hu
6. Zheng Zhang
7. Fangyun Wei
8. Avinash Ravichandran

Citations in the corpus

• Transformer-Based Visual Segmentation: A Survey (2023, methodology): “Besides these open vocabulary detection settings, recent works [248], [249], [280], [281], [282] perform open vocabulary segmentation.”
• Side Adapter Network for Open-Vocabulary Semantic Segmentation (2023, methodology): “† SimSeg [33] trained with a subset of COCO Stuff in their paper.”
• WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation (2023, positive): “Without additional fine-tuning, text prompts can then be used to extract knowledge from such models for zero-/few-shot transfer to downstream tasks including image classification [27], object detection [11] and segmentation [45].”
• Vision-Language Models for Vision Tasks: A Survey (2023, background): “For example, [35], [180], [181] achieve open-vocabulary semantic segmentation by first class-agnostic segmentation by grouping pixels into multiple segments and then segment recognition with CLIP.”