DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting

By Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, Jiwen Lu. 2021

830 citations recorded by the source corpus.

Recent progress has shown that large-scale pre-training using contrastive image-text pairs can be a promising alternative for high-quality visual representation learning from natural language supervision. Benefiting from a broader source of supervision, this new paradigm exhibits impressive transferability to downstream classification tasks and datasets. However, the problem of transferring the knowledge learned from image-text pairs to more complex dense prediction tasks has barely been visited. In this work, we present a new framework for dense prediction by implicitly and explicitly leveraging the pre-trained knowledge from CLIP. Specifically, we convert the original image-text matching problem in CLIP to a pixel-text matching problem and use the pixel-text score maps to guide the learning of dense prediction models. By further using the contextual information from the image to prompt the language model, we are able to facilitate our model to better exploit the pretrained knowledge. Our method is model-agnostic, which can be applied to arbitrary dense prediction systems and various pre-trained visual backbones including both CLIP models and ImageNet pre-trained models. Extensive experiments demonstrate the superior performance of our methods on semantic segmentation, object detection, and instance segmentation tasks. Code is available at https://github.com/raoyongming/DenseCLIP.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Prompting & Reasoning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Prompting & Reasoning

In Multimodal Learning

Citations in the corpus

A Simple Framework for Open-Vocabulary Segmentation and Detection (2023, methodology): “Apart from using foundation models, DenseCLIP [41] and GroupViT [46] show that fine-tuning from a foundation model or training from scratch can also yield superior zero-shot performance.”
Generalized Decoding for Pixel, Image, and Language (2022, methodology): “Instead of using existing models, GroupViT [81] performed language-image pretraining from scratch with a bottom-up grouping ViT [19], while DenseCLIP [65] demonstrated the superiority of foundation models in finetuning settings compared with supervised models.”
Conditional Prompt Learning for Vision-Language Models (2022, background): “In computer vision, prompt learning is a nascent research direction that has only been explored very recently [27, 42, 56,58,62].”
CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection (2023, background): “[57] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu.”
Segment Everything Everywhere All at Once (2023, methodology): “Many open-vocabulary segmentation models have been proposed recently [24, 13, 18, 10, 43, 51], which employ large pretrained vision-language models like CLIP [20] to transfer visual-semantic knowledge by freeze or fine-tune their weights.”
Decomposing NeRF for Editing via Feature Field Distillation (2022, background): “Recent development of image encoder architectures [96, 22, 71] and large-scale training [69, 12] have improved the ability and generalization of vision models, including zero-shot models [44, 52, 99, 103, 114, 72].”
OpenMask3D: Open-Vocabulary 3D Instance Segmentation (2023, background): “As CLIP gained popularity for image classification, numerous new approaches [18, 20, 23, 29, 36, 39, 41, 44, 53, 65, 68, 69] have emerged to tackle open-vocabulary image semantic segmentation.”
AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection (2023, positive): “Benefiting from the glocal context optimization, AnomalyCLIP can capture local anomaly semantics such that we can simultaneously perform classification and segmentation tasks without an additional decoder network like Rao et al. (2022).”
CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Alignment (2022, methodology): “…success of the CLIP has demonstrated its unprecedented power on varies downstream tasks, including vision understanding (Gu et al., 2021; Wang et al., 2021a; Rao et al., 2022), image-text generation (Patashnik et al., 2021; Mokady et al., 2021) and so on (Guzhov et al., 2022; Zhang et al., 2022).”
Expanding Language-Image Pretrained Models for General Video Recognition (2022, methodology): “For example, CoOp [61], CLIP-Adapter [16] and Tip-Adapter [58] use the pretrained CLIP for improving the few-shot transfer, while PointCLIP [59] and DenseCLIP [38,60] transfer the knowledge to point cloud understanding and dense prediction, respectively.”
Unleashing Text-to-Image Diffusion Models for Visual Perception (2023, neutral): “visial-language pre-training CLIP-ViT-B [42] 80K 640(2) 340G 50.”
LMSeg: Language-guided Multi-dataset Segmentation (2023, methodology): “DenseCLIP (Rao et al., 2022) applies CLIP to dense prediction tasks via language-guided ﬁne-tuning and achieves promising results.”
Extract Free Dense Labels from CLIP (2021, background): “A concurrent work, DenseCLIP [46], aims to better fine-tune the CLIP pre-trained weights on target semantic segmentation datasets without keeping the zero-shot transferability, which are different from our setting.”
Bridging CLIP and StyleGAN through Latent Alignment for Image Editing (2022, neutral): “In recent years, vision-language pre-training [14, 26, 29, 36, 47, 48] has become an attractive novel research topic.”
Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners (2023, background): “Some efforts [37,59,78,82] utilize learnable textual inputs and optimize them during training.”