Unsupervised Prompt Learning for Vision-Language Models

By Hao Huang, J.-L. Chu, Fangyun Wei. 2022

182 citations recorded by the source corpus.

Contrastive vision-language models like CLIP have shown great progress in transfer learning. In the inference stage, the proper text description, also known as prompt, needs to be carefully designed to correctly classify the given images. In order to avoid laborious prompt engineering, recent works such as CoOp, CLIP-Adapter and Tip-Adapter propose to adapt vision-language models for downstream image recognition tasks on a small set of labeled data. Though promising improvements are achieved, requiring labeled data from the target datasets may restrict the scalability. In this paper, we explore a different scenario, in which the labels of the target datasets are unprovided, and we present an unsupervised prompt learning (UPL) approach to avoid prompt engineering while simultaneously improving transfer performance of CLIP-like vision-language models. As far as we know, UPL is the first work to introduce unsupervised learning into prompt learning. Experimentally, our UPL outperforms original CLIP with prompt engineering on ImageNet as well as other 10 datasets. An enhanced version of UPL is even competitive with the 8-shot CoOp and the 8-shot TIP-Adapter on most datasets. Code and models are available at https://github.com/tonyhuang2022/UPL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning (2022, methodology): “There have been emerging a bunch of prompting methods, such as [35, 38, 26, 76, 77, 3, 24, 66, 14], to name a few 6 .”
MaPLe: Multi-modal Prompt Learning (2022, background): “To address the above challenges, existing works propose language prompt learning to avoid manually adjusting the prompt templates and providing a mechanism to adapt the model while keeping the original weights frozen [14, 25, 29, 48, 49].”
Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners (2023, background): “Some efforts [37,59,78,82] utilize learnable textual inputs and optimize them during training.”
What does a platypus look like? Generating customized prompts for zero-shot image classification (2022, methodology): “These methods learn prompts formats which yield high accuracy for image clas-siﬁcation using either supervised [66, 43] or unsupervised [22] methods.”
Self-regulating Prompts: Foundational Model Adaptation without Forgetting (2023, result): “Prompt learning has emerged as a more efficient alternative to fine-tuning large-scale models, as shown in recent studies [58, 59, 3, 17, 40, 28].”
Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models (2022, background): “Another line of work proposes to tune the prompt in an unsupervised manner [31, 32].”
Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “Third, unsupervised transfer can perform comparably with few-shot supervised transfer ( e.g. , unsupervised UPL [143] outperforms 2-shot supervised CoOp [31] by 0.4%, unsupervised TPT [144] is comparable with 16-shot CoOp [31]), largely because unsupervised transfer can access massive unlabelled…”