Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP

By Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Péter Vajda, Diana Marculescu. Computer Vision and Pattern Recognition, 2022

767 citations recorded by the source corpus.

Open-vocabulary semantic segmentation aims to segment an image into semantic regions according to text descriptions, which may not have been seen during training. Recent two-stage methods first generate class-agnostic mask proposals and then leverage pre-trained vision-language models, e.g., CLIP, to classify masked regions. We identify the performance bottleneck of this paradigm to be the pre-trained CLIP model, since it does not perform well on masked images. To address this, we propose to finetune CLIP on a collection of masked image regions and their corresponding text descriptions. We collect training data by mining an existing image-caption dataset (e.g., COCO Captions), using CLIP to match masked image regions to nouns in the image captions. Compared with the more precise and manually annotated segmentation labels with fixed classes (e.g., COCO-Stuff), we find our noisy but diverse dataset can better retain CLIP's generalization ability. Along with finetuning the entire model, we utilize the “blank” areas in masked images using a method we dub mask prompt tuning. Experiments demonstrate mask prompt tuning brings significant improvement without modifying any weights of CLIP, and it can further improve a fully finetuned model. In particular, when trained on COCO and evaluated on ADE20K-150, our best model achieves 29.6% mIoU, which is +8.5% higher than the previous state-of-the-art. For the first time, open-vocabulary generalist models match the performance of supervised specialist models in 2017 without dataset specific adaptations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/29c2d3d77b6d6f24f4356d5ba20c1a6ab4229c76

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

ConceptFusion: Open-set Multimodal 3D Mapping (2023, methodology): “segmentation datasets, while OpenSeg [18], OVSeg [26], and RegionCLIP [23] use image-caption datasets and employ region-word grounding.”
Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models (2023, methodology): “It has been employed in various downstream applications, from image-based object detection and segmentation [13, 14] to 3D applications [15, 16].”
LERF: Language Embedded Radiance Fields (2023, background): “Con-Volume sequently, as noted by the authors of OV-Seg [22], such networks often face difﬁculties in accurately segmenting unlabeled hierarchical components of the original masks, such as object parts.”
LangSplat: 3D Language Gaussian Splatting (2023, methodology): “We observe that our method not only out-performs 2D-based methods such as ODISE [41] and OV-Seg [22], but also achieves better results than 3D-based methods including LERF [18] and 3D-OVS [23] by a large margin.”
Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “OVSeg [190] [code] Adapt CLIP with the region-word pairs generated by the modified MaskFormer.”
Side Adapter Network for Open-Vocabulary Semantic Segmentation (2023, methodology): “We also adopt the data augmentation [7,22,33] with random image resizing in the short-side range of [320,1024] and a crop size of 640(2).”
RSPrompter: Learning to Prompt for Remote Sensing Instance Segmentation Based on Visual Foundation Model (2023, background): “Currently, research on foundational segmentation and detection models has become a popular area of focus [33], [49], [50], [51], [52], [53].”
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, methodology): “Recent success of 2D open-vocabulary segmentation models such as LSeg [43], OpenSeg [20], and OV-Seg[45] has motivated researchers in the field of 3D scene understanding to explore the open vocabulary setting [6, 12, 19, 24, 30, 32, 35, 37, 49, 52, 59, 60].”