Test-Time Prompt Tuning for Zero-Shot Generalization in Vision-Language Models

By Manli Shu, Weili Nie, De-An Huang, Zhiding Yu, Tom Goldstein, Anima Anandkumar, Chaowei Xiao. Neural Information Processing Systems, 2022

552 citations recorded by the source corpus.

Pre-trained vision-language models (e.g., CLIP) have shown promising zero-shot generalization in many downstream tasks with properly designed text prompts. Instead of relying on hand-engineered prompts, recent works learn prompts using the training data from downstream tasks. While effective, training on domain-specific data reduces a model's generalization capability to unseen new domains. In this work, we propose test-time prompt tuning (TPT), a method that can learn adaptive prompts on the fly with a single test sample. For image classification, TPT optimizes the prompt by minimizing the entropy with confidence selection so that the model has consistent predictions across different augmented views of each test sample. In evaluating generalization to natural distribution shifts, TPT improves the zero-shot top-1 accuracy of CLIP by 3.6% on average, surpassing previous prompt tuning approaches that require additional task-specific training data. In evaluating cross-dataset generalization with unseen categories, TPT performs on par with the state-of-the-art approaches that use additional training data. Project page: https://azshue.github.io/TPT.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/12ad6e798487223f4c17aac69c9853bca8bc7830

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Prompt-aligned Gradient for Prompt Tuning (2022, methodology): “TPT [46] proposes a test-time prompt tuning, which does not require training data and optimizes the prompt to achieve consistent predictions across different augmented views.”
Towards Out-Of-Distribution Generalization: A Survey (2021, background): “[237] introduce an approach to learn adaptive prompts on the fly using a single test sample.”
MaPLe: Multi-modal Prompt Learning (2022, background): “To address the above challenges, existing works propose language prompt learning to avoid manually adjusting the prompt templates and providing a mechanism to adapt the model while keeping the original weights frozen (Zhou et al., 2022a;b; Lu et al., 2022; Huang et al., 2022; Manli et al., 2022).”
Vision-Language Models for Vision Tasks: A Survey (2023, background): “…comparably with few-shot supervised transfer ( e.g. , unsupervised UPL [143] outperforms 2-shot supervised CoOp [31] by 0.4%, unsupervised TPT [144] is comparable with 16-shot CoOp [31]), largely because unsupervised transfer can access massive unlabelled downstream data with much lower…”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, background): “TPT (test-time prompt tuning) [161] studies prompt tuning on the fly without additional training samples.”
Self-regulating Prompts: Foundational Model Adaptation without Forgetting (2023, methodology): “(Middle) : Averaged across 11 image recognition datasets, PromptSRC surpasses existing methods on the base-to-novel generalization setting.”