What does a platypus look like? Generating customized prompts for zero-shot image classification

By Sarah Pratt, Rosanne Liu, Ali Farhadi. IEEE International Conference on Computer Vision, 2022

367 citations recorded by the source corpus.

Open-vocabulary models are a promising new paradigm for image classification. Unlike traditional classification models, open-vocabulary models classify among any arbitrary set of categories specified with natural language during inference. This natural language, called "prompts", typically consists of a set of hand-written templates (e.g., "a photo of a {}") which are completed with each of the category names. This work introduces a simple method to generate higher accuracy prompts, without relying on any explicit knowledge of the task domain and with far fewer hand-constructed sentences. To achieve this, we combine open-vocabulary models with large language models (LLMs) to create Customized Prompts via Language models (CuPL, pronounced "couple"). In particular, we leverage the knowledge contained in LLMs in order to generate many descriptive sentences that contain important discriminating characteristics of the image categories. This allows the model to place a greater importance on these regions in the image when making predictions. We find that this straightforward and general approach improves accuracy on a range of zero-shot image classification benchmarks, including over one percentage point gain on ImageNet. Finally, this simple baseline requires no additional training and remains completely zero-shot. Code available at https://github.com/sarahpratt/CuPL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/007fe51fddfae031570f2d69e67dc67e1fe33621

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

SpatialRGPT: Grounded Spatial Reasoning in Vision Language Model (2024, background): “Despite the impressive advancements in Vision Language Models (VLMs) across a variety of tasks such as image classification [4], captioning [5], object detection [6], and document parsing [7], etc., these models still face significant challenges with spatial reasoning.”
Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “Second, the performance of few-shot supervised transfer lag far behind that of supervised transfer ( e.g. , 87.1% in WiseFT [162] and 76.6% in CuPL [160]), largely because VLMs may overfit to few-shot labelled samples with degraded generalization.”
Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunities (2024, background): “[129] Using LLM’s inherent knowledge to generate descriptive sentences with crucial discriminating characteristics of the image categories.”
Visual Classification via Description from Large Language Models (2022, background): “We encourage readers to see concurrent work (Pratt et al., 2022) that demonstrates that prompt ensembling with prompts obtained by large language models can improve accuracy on recognition tasks.”
Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners (2023, methodology): “Our CaFo refers to CuPL [55] to produce semantic-rich texts to prompt CLIP for better text-image alignment.”
Better Language Models Exhibit Higher Visual Alignment (2024, unknown): “For AWA2 and CUB, we use splits by Xian et al. (2017) while randomly dividing aircraft types into 50 seen and 20 unseen classes.”