Multiresolution Knowledge Distillation for Anomaly Detection

By Mohammadreza Salehi, Niousha Sadjadi, Soroosh Baselizadeh, Mohammad Hossein Rohban, Hamid Reza Rabiee. Computer Vision and Pattern Recognition, 2020

663 citations recorded by the source corpus.

Unsupervised representation learning has proved to be a critical component of anomaly detection/localization in images. The challenges to learn such a representation are two-fold. Firstly, the sample size is not often large enough to learn a rich generalizable representation through conventional techniques. Secondly, while only normal samples are available at training, the learned features should be discriminative of normal and anomalous samples. Here, we propose to use the "distillation" of features at various layers of an expert network, which is pre-trained on ImageNet, into a simpler cloner network to tackle both issues. We detect and localize anomalies using the discrepancy between the expert and cloner networks’ intermediate activation values given an input sample. We show that considering multiple intermediate hints in distillation leads to better exploitation of the expert’s knowledge and a more distinctive discrepancy between the two networks, compared to utilizing only the last layer activation values. Notably, previous methods either fail in precise anomaly localization or need expensive region-based training. In contrast, with no need for any special or intensive training procedure, we incorporate interpretability algorithms in our novel framework to localize anomalous regions. Despite the striking difference between some test datasets and ImageNet, we achieve competitive or significantly superior results compared to SOTA on MNIST, F-MNIST, CIFAR-10, MVTecAD, Retinal-OCT, and two other medical datasets on both anomaly detection and localization.

Subfields: Artificial Intelligence, Epidemiology, Radiology, Nuclear Medicine and Imaging

View Paper: https://www.semanticscholar.org/paper/6885c45614f78f9d2e7cc8ef11b3c38b34e67f7d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Epidemiology

In Radiology, Nuclear Medicine and Imaging

Citations in the corpus

AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection (2023, neutral): “…(Jha et al., 2020), and Endo (Hicks et al., 2021), thyroid nodule detection dataset TN3k (Gong et al., 2021), brain tumor detection datasets HeadCT (Salehi et al., 2021), BrainMRI (Salehi et al., 2021), Br35H (Hamada., 2020), and COVID-19 detection dataset COVID-19 (Chowdhury et al., 2020; Rahman…”
Generalized Out-of-Distribution Detection: A Survey (2021, background): “[87], [88], [89], [89], [90], [91], [92], [93], [94], [95], [96], [97], [98]”
SimpleNet: A Simple Network for Image Anomaly Detection and Localization (2023, methodology): “The baselines include VAE [2], LSA [1], DSVDD [25], OCGAN [19], HRN [15], AnoGAN [27], DAAD [14], MKD [26], DisAug CLR [28], IGD [5] and RevDist [7].”
Deep Industrial Image Anomaly Detection: A Survey (2023, neutral): “While STPM [11] and MKD [10] both use multi-scale features under different network layers for distillation, they do so in different ways.”
Revisiting Reverse Distillation for Anomaly Detection (2023, background): “Thus, it is expected to learn the distribution of normal samples, subsequently generating the out-of-distribution representations when inference with anomalous query [30].”
Anomaly Detection via Reverse Distillation from One-Class Embedding (2022, methodology): “In conventional KD, the student network adopts a similar or identical neural network to the teacher model, accepts raw data/images as input, and targets to match its feature activations to the teacher’s [4, 33].”
Towards Total Recall in Industrial Anomaly Detection (2021, methodology): “For the majority of these approaches, anomaly localization comes naturally based on pixel-wise reconstruction errors, saliency-based approaches such as GradCAM [47] or XRAI [28] can be used for anomaly segmentation [42,45,52] as well.”
DeSTSeg: Segmentation Guided Denoising Student-Teacher for Anomaly Detection (2022, background): “[32], and [27] adopted multi-level feature representation alignment to capture both low-level and high-level anomalies.”
EfficientAD: Accurate Visual Anomaly Detection at Millisecond-Level Latencies (2023, background): “[56] Mohammadreza Salehi, Niousha Sadjadi, Soroosh Baselizadeh, Mohammad H.”