Adversarial vulnerability of powerful near out-of-distribution detection

By Stanislav Fort. 2022

18 citations recorded by the source corpus.

There has been a significant progress in detecting out-of-distribution (OOD) inputs in neural networks recently, primarily due to the use of large models pretrained on large datasets, and an emerging use of multi-modality. We show a severe adversarial vulnerability of even the strongest current OOD detection techniques. With a small, targeted perturbation to the input pixels, we can change the image assignment from an in-distribution to an out-distribution, and vice versa, easily. In particular, we demonstrate severe adversarial vulnerability on the challenging near OOD CIFAR-100 vs CIFAR-10 task, as well as on the far OOD CIFAR-100 vs SVHN. We study the adversarial robustness of several post-processing techniques, including the simple baseline of Maximum of Softmax Probabilities (MSP), the Mahalanobis distance, and the newly proposed \textit{Relative} Mahalanobis distance. By comparing the loss of OOD detection performance at various perturbation strengths, we demonstrate the beneficial effect of using ensembles of OOD detectors, and the use of the \textit{Relative} Mahalanobis distance over other post-processing methods. In addition, we show that even strong zero-shot OOD detection using CLIP and multi-modality suffers from a severe lack of adversarial robustness as well. Our code is available at https://github.com/stanislavfort/adversaries_to_OOD_detection

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Luke S. Zettlemoyer
2. Jared Kaplan
3. Percy Liang
4. Yejin Choi
5. Quoc V. Le
6. Sam McCandlish
7. Danny Hernandez
8. Mohit Bansal

In Computer Vision and Pattern Recognition

1. T. Henighan
2. Amanda Askell
3. Saurav Kadavath
4. Dawn Drain
5. Neel Nanda
6. Jared Kaplan
7. Catherine Olsson
8. Jack Clark

In Electrical and Electronic Engineering

1. Jack Clark
2. Dario Amodei

In Safety, Robustness & Interpretability

1. Saurav Kadavath
2. Neel Nanda
3. Jack Clark
4. Dario Amodei
5. Stanislav Fort
6. Tom B. Brown
7. Danny Hernandez

Citations in the corpus

• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback (2022, methodology): “The more difficult case of OOD detection, the so-called near-OOD detection [Winkens et al., 2020], has recently been improved upon significantly using pre-training and large models [Fort et al., 2021].”