Zero-Shot Out-of-Distribution Detection Based on the Pre-trained Model CLIP

By Sepideh Esmaeilpour, Bing Liu, Eric Robertson, Lei Shu. 2021

229 citations recorded by the source corpus.

In an out-of-distribution (OOD) detection problem, samples of known classes (also called in-distribution classes) are used to train a special classifier. In testing, the classifier can (1) classify the test samples of known classes to their respective classes and also (2) detect samples that do not belong to any of the known classes (i.e., they belong to some unknown or OOD classes). This paper studies the problem of zero-shot out-of-distribution (OOD) detection, which still performs the same two tasks in testing but has no training except using the given known class names. This paper proposes a novel and yet simple method (called ZOC) to solve the problem. ZOC builds on top of the recent advances in zero-shot classification through multi-modal representation learning. It first extends the pre-trained language-vision model CLIP by training a text-based image description generator on top of CLIP. In testing, it uses the extended model to generate candidate unknown class names for each test sample and computes a confidence score based on both the known class names and candidate unknown class names for zero-shot OOD detection. Experimental results on 5 benchmark datasets for OOD detection demonstrate that ZOC outperforms the baselines by a large margin.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zixuan Ke
2. Bin Liu
3. Gyuhak Kim
4. Tatsuya Konishi
5. Bing Liu
6. Sahisnu Mazumder
7. Changnan Xiao
8. Eric Robertson

In Radiology, Nuclear Medicine and Imaging

1. Tatsuya Konishi
2. Gyuhak Kim
3. Changnan Xiao
4. Matthew P. Lungren
5. Bing Liu
6. Bin Liu
7. Zixuan Ke

Citations in the corpus

• A Theoretical Study on Solving Continual Learning (2022, methodology): “Many OOD detection algorithms can perform both IND prediction and OOD detection [6, 7, 8, 9].”
• Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing (2023, methodology): “For this step, a separate BioViL-T is trained on ‘clean’ studies with single acquisitions and later used in a zero-shot setting to detect out-of-distribution samples [26,27] arising from the re-imaging process.”
• ZegCLIP: Towards Adapting CLIP for Zero-shot Semantic Segmentation (2022, methodology): “Recently, the pretrained vision-language model CLIP [36] has been adopted into various dense prediction tasks, such as referring segmentation [43], semantic segmentation [33], and detection [14].”
• AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection (2023, unknown): “CLIP-AD (Liznerski et al., 2022) and ZOC (Esmaeilpour et al., 2022) are early studies in utilizing CLIP for ZSAD, but they mainly focus on the anomaly classification task.”
• Self-Initiated Open World Learning for Autonomous AI Agents (2021, background): “Recently, zero-shot out-of-distribution detection is investigated in (Esmaeilpour et al. 2022) based on the pre-trained model CLIP (Radford et al. 2021).”
• Generalized Out-of-Distribution Detection: A Survey (2021, unknown): “For the classification task itself, the researchers also extended the OOD detection technique to improve the reliability of zero-shot pretrained models (Esmaeilpour et al., 2022) (e.g.,CLIP).”