Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classification
By Yue Yang, Artemis Panagopoulou, Shenghao Zhou, Daniel Jin, Chris Callison-Burch, Mark Yatskar. Computer Vision and Pattern Recognition, 2022
386 citations recorded by the source corpus.
Concept Bottleneck Models (CBM) are inherently interpretable models that factor model decisions into humanreadable concepts. They allow people to easily understand why a model is failing, a critical feature for high-stakes applications. CBMs require manually specified concepts and often under-perform their black box counterparts, preventing their broad adoption. We address these shortcomings and are first to show how to construct high-performance CBMs without manual specification of similar accuracy to black box models. Our approach, Language Guided Bottlenecks (LaBo), leverages a language model, GPT-3, to define a large space of possible bottlenecks. Given a problem domain, LaBo uses GPT-3 to produce factual sentences about categories to form candidate concepts. LaBo efficiently searches possible bottlenecks through a novel submodular utility that promotes the selection of discriminative and diverse information. Ultimately, GPT-3's sentential concepts can be aligned to images using CLIP, to form a bottleneck layer. Experiments demonstrate that LaBo is a highly effective prior for concepts important to visual recognition. In the evaluation with 11 diverse datasets, LaBo bottlenecks excel at few-shot classification: they are 11.7% more accurate than black box linear probes at 1 shot and comparable with more data. Overall, LaBo demonstrates that inherently interpretable models can be widely applied at similar, or better, performance than black box approaches.11Code and data are available at https://github.com/YueYANG1996/LaBo
Subfields: Artificial Intelligence
View Paper: https://www.semanticscholar.org/paper/6d1badc223cfca6ae23fcce811b9dbcb8645cdef
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Rosanne Liu
- Ali Farhadi
- Sarah Pratt
- Yejin Choi
- Mohit Bansal
- Luke S. Zettlemoyer
- Abhinav Kumar Gupta
- Aniruddha Kembhavi
Citations in the corpus
- What does a platypus look like? Generating customized prompts for zero-shot image classification (2022, methodology): “Additionally, [61] use an LLM to generate a list of natural language attributes for ImageNet classes and then select a subset of these attributes for each class in a few-shot manner.”
- Better Language Models Exhibit Higher Visual Alignment (2024, unknown): “We report the average per-class classification accuracy across the datasets as a measure of visual generalization ability facilitated by the language embeddings.”