Extract Free Dense Labels from CLIP

By Chong Zhou, Chen Change Loy, Bo Dai. European Conference on Computer Vision, 2021

806 citations recorded by the source corpus.

Contrastive Language-Image Pre-training (CLIP) has made a remarkable breakthrough in open-vocabulary zero-shot image recognition. Many recent studies leverage the pre-trained CLIP models for image-level classification and manipulation. In this paper, we wish examine the intrinsic potential of CLIP for pixel-level dense prediction, specifically in semantic segmentation. To this end, with minimal modification, we show that MaskCLIP yields compelling segmentation results on open concepts across various datasets in the absence of annotations and fine-tuning. By adding pseudo labeling and self-training, MaskCLIP+ surpasses SOTA transductive zero-shot semantic segmentation methods by large margins, e.g., mIoUs of unseen classes on PASCAL VOC/PASCAL Context/COCO Stuff are improved from 35.6/20.7/30.3 to 86.1/66.7/54.7. We also test the robustness of MaskCLIP under input corruption and evaluate its capability in discriminating fine-grained objects and novel concepts. Our finding suggests that MaskCLIP can serve as a new reliable source of supervision for dense prediction tasks to achieve annotation-free segmentation. Source code is available at https://github.com/chongzhou96/MaskCLIP.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/0a7e7347e16bf13d710f6f3d30748baabdbb96ad

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

OpenScene: 3D Scene Understanding with Open Vocabularies (2022, background): “Recent work has shown that these models can be used to increase the flexibility and generalizability of 2D image semantic segmentation [17, 30, 44, 57, 59, 60].”
Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation (2023, neutral): “Feature adaptation methods [5, 16, 33, 62, 64, 65] can further pivot VFMs towards different tasks.”
ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, methodology): “Besides, [56] utilizes CLIP to generate pseudo labels of novel classes for self-training but will be invalid if the unseen class names in inference are unknown in the training stage (“inductive” zero-shot setting).”
Learning Customized Visual Models with Retrieval-Augmented Knowledge (2023, methodology): “Following MaskCLIP [87], we use ViT-B/16 checkpoints, and use their ofﬁcial code base to train and evaluate models.”
Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “MaskCLIP [163] extracts dense image features by modifying the architecture of the CLIP image encoder.”
WinCLIP: Zero-/Few-Shot Anomaly Classification and Segmentation (2023, positive): “Second, MaskCLIP [55] is a general semantic segmentation model based on CLIP.”
F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models (2022, methodology): “The leading DetPro (Du et al., 2022) shows the effectiveness of prompt optimization (Zhou et al., 2022b) which can potentially benefit F-VLM too.”
Foundation models in robotics: Applications, challenges, and the future (2023, neutral): “In Table II some robotic-speciﬁc foundation models are re-ported along with information about their size and architecture, pretrained task, inference time, and hardware setup.”