Doubly Right Object Recognition: A Why Prompt for Visual Rationales

By Chengzhi Mao, Revant Teotia, Amrutha Sundar, Sachit Menon, Junfeng Yang, Xin Eric Wang, Carl Vondrick. 2022

40 citations recorded by the source corpus.

Many visual recognition models are evaluated only on their classification accuracy, a metric for which they obtain strong performance. In this paper, we investigate whether computer vision models can also provide correct rationales for their predictions. We propose a “doubly right” object recognition benchmark, where the metric requires the model to simultaneously produce both the right labels as well as the right rationales. We find that state-of-the-art visual models, such as CLIP, often provide incorrect rationales for their categorical predictions. However, by transferring the rationales from language models into visual representations through a tailored dataset, we show that we can learn a “why prompt,” which adapts large visual representations to produce correct rationales. Visualizations and empirical experiments show that our prompts significantly improve performance on doubly right object recognition, in addition to zero-shot transfer to unseen tasks and datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Vinay Kumar Verma
2. Riddhiman Moulick
3. Abir Das
4. Saptarshi Ghosh
5. Anurag Roy
6. Carl Vondrick
7. D'idac Sur'is
8. Sachit Menon

In Computer Vision and Pattern Recognition

1. Vinay Kumar Verma
2. Riddhiman Moulick
3. Abir Das
4. Saptarshi Ghosh
5. Anurag Roy
6. D'idac Sur'is
7. Sachit Menon
8. Carl Vondrick

In Safety, Robustness & Interpretability

1. Vinay Kumar Verma
2. Carl Vondrick
3. Sachit Menon

Citations in the corpus

• ViperGPT: Visual Inference via Python Execution for Reasoning (2023, background): “Other recent works [63, 45, 55, 35, 37, 15] show that large pretrained models can be used together to great effect, but hand-specify the particular way models are combined.”
• Convolutional Prompting meets Language Models for Continual Learning (2024, methodology): “Inspired by works like [30, 32], the class attributes are generated by using the query – “ What are useful features for distinguishing a [class name] in a photo?”