Visual Classification via Description from Large Language Models

By Sachit Menon, Carl Vondrick. 2022

411 citations recorded by the source corpus.

Vision-language models (VLMs) such as CLIP have shown promising performance on a variety of recognition tasks using the standard zero-shot classification procedure -- computing similarity between the query image and the embedded words for each category. By only using the category name, they neglect to make use of the rich context of additional information that language affords. The procedure gives no intermediate understanding of why a category is chosen, and furthermore provides no mechanism for adjusting the criteria used towards this decision. We present an alternative framework for classification with VLMs, which we call classification by description. We ask VLMs to check for descriptive features rather than broad categories: to find a tiger, look for its stripes; its claws; and more. By basing decisions on these descriptors, we can provide additional cues that encourage using the features we want to be used. In the process, we can get a clear idea of what features the model uses to construct its decision; it gains some level of inherent explainability. We query large language models (e.g., GPT-3) for these descriptors to obtain them in a scalable way. Extensive experiments show our framework has numerous advantages past interpretability. We show improvements in accuracy on ImageNet across distribution shifts; demonstrate the ability to adapt VLMs to recognize concepts unseen during training; and illustrate how descriptors can be edited to effectively mitigate bias compared to the baseline.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yu Qiao
2. Hongsheng Li
3. Peng Gao
4. Renrui Zhang
5. Ali Farhadi
6. Siyuan Huang
7. Shijian Lu
8. Vinay Kumar Verma

In Computer Vision and Pattern Recognition

1. Hongsheng Li
2. Yu Qiao
3. Peng Gao
4. Xiaogang Wang
5. Luc Van Gool
6. Dacheng Tao
7. Renrui Zhang
8. Ali Farhadi

In Large Language Models

1. Ali Farhadi
2. Yu Qiao
3. Peng Gao
4. Sheng Jin
5. Shijian Lu
6. Jingyi Zhang
7. Jiaxing Huang
8. Renrui Zhang

Citations in the corpus

• Vision-Language Models for Vision Tasks: A Survey (2023, methodology): “Different from prompt engineering and prompt learning, several attempts [160], [161] exploit LLMs [172] to generate text prompts that better describe downstream tasks.”
• What does a platypus look like? Generating customized prompts for zero-shot image classification (2022, methodology): “[33] also explores LLM generated descriptions for image classification.”
• Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification (2022, neutral): “Our model surpasses end-to-end models in few shot scenarios and achieves comparable performance in large data settings, while concurrent work [35] only evaluates on zero-shot settings.”
• Convolutional Prompting meets Language Models for Continual Learning (2024, methodology): “Inspired by works like [30, 32], the class attributes are generated by using the query – “ What are useful features for distinguishing a [class name] in a photo?”
• Prompt, Generate, Then Cache: Cascade of Foundation Models Makes Strong Few-Shot Learners (2023, background): “Other works [50, 56] propose to leverage linguistic knowledge pre-trained from large language models to generate prompts for each visual category, which enhances vision-language models without any additional training or labeling.”
• Better Language Models Exhibit Higher Visual Alignment (2024, unknown): “We report the average per-class classification accuracy across the datasets as a measure of visual generalization ability facilitated by the language embeddings.”