Deep Structured Energy Based Models for Anomaly Detection

By Shuangfei Zhai, Yu Cheng, Weining Lu, Zhongfei Zhang. 2016

463 citations recorded by the source corpus.

In this paper, we attack the anomaly detection problem by directly modeling the data distribution with deep architectures. We propose deep structured energy based models (DSEBMs), where the energy function is the output of a deterministic deep neural network with structure. We develop novel model architectures to integrate EBMs with different types of data such as static data, sequential data, and spatial data, and apply appropriate model architectures to adapt to the data structure. Our training algorithm is built upon the recent development of score matching \cite{sm}, which connects an EBM with a regularized autoencoder, eliminating the need for complicated sampling method. Statistically sound decision criterion can be derived for anomaly detection purpose from the perspective of the energy landscape of the data distribution. We investigate two decision criteria for performing anomaly detection: the energy score and the reconstruction error. Extensive empirical studies on benchmark tasks demonstrate that our proposed model consistently matches or outperforms all the competing methods.

Subfields: Artificial Intelligence, Computer Networks and Communications, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Representation & Self-Supervised Learning

Citations in the corpus

Flow Contrastive Estimation of Energy-Based Models (2019, background): “Another line of work is to estimate the parameters of EBM by score matching [26, 74, 63, 64].”
Maximum Entropy Generators for Energy-Based Models (2019, methodology): “Our baseline for this task is Deep Structured Energy-based Model for Anomaly Detection (DSEBM) (Zhai et al., 2016), which trains deep energy models such as Convolutional and Recurrent EBMs using denoising score matching instead of maximum likelihood, for performing anomaly detection.”
Learning Memory-Guided Normality for Anomaly Detection (2020, methodology): “Reconstructive models encode normal patterns using representation learning methods such as an AE [48, 36], a sparse dictionary learning [6, 49, 24], and a generative model [43].”
CutPaste: Self-Supervised Learning for Anomaly Detection and Localization (2021, background): “Anomaly detection under one-class classification setting, where we assume only the normal data is given during the training, has been widely studied [49, 56, 46, 63, 66, 13, 42, 36, 27].”
Learning Energy-Based Models in High-Dimensional Spaces with Multiscale Denoising-Score Matching (2019, background): “, 2010), outlier/novelty detection (Zhai et al., 2016; Choi and Jang, 2018), sample generation (Nijkamp et al.”
End-to-End Learning for Structured Prediction Energy Networks (2017, unknown): “Recently, Zhai et al. (2016) trained energy-based density models for anomaly detection by exploiting the connections be-tween denosing autoencoders, energy-based models, and score matching (Vincent, 2011).”
CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances (2020, background): “Most recent works can be grouped into four categories: (a) density-based [66, 43, 7, 12, 52, 57, 17], (b) reconstruction-based [55, 68, 10, 50, 48, 8], (c) one-class classiﬁer [56, 53], and (d) self-supervised [15, 26, 1] methods.”
Memorizing Normality to Detect Anomaly: Memory-Augmented Deep Autoencoder for Unsupervised Anomaly Detection (2019, methodology): “For example, structured energy based deep neural network [43] is used to model the training samples.”
Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection (2018, methodology): “Deep structured energy based model (DSEBM) (Zhai et al. (2016)) is a state-of- the-art deep learning method for unsupervised anomaly detection.”
Set Prediction without Imposing Structure as Conditional Density Estimation (2020, methodology): “In this work we focus on distributions taking the form of deep energy-based models (Ngiam et al., 2011; Zhai et al., 2016; Belanger & McCallum, 2016): with Z as the partition function and E θ the energy function with parameters θ .”
Fully Convolutional Cross-Scale-Flows for Image-based Defect Detection (2021, unknown): “[41] combine energy-based models and regularized autoencoders to model the data distribution.”
A Unifying Review of Deep and Shallow Anomaly Detection (2020, methodology): “This sort of model has been successfully used for deep AD [271].”
A Deep Neural Network for Unsupervised Anomaly Detection and Diagnosis in Multivariate Time Series Data (2018, methodology): “Besides traditional methods, deep learning based unsupervised anomaly detection algorithms (Malhotra et al. 2016; Zhai et al. 2016; Zhou and Paffenroth 2017; Zong et al. 2018) have gained a lot attention recently.”
Learning Neural Random Fields with Inclusive Auxiliary Generators (2018, methodology): “Anomaly detection has been extensively studied, as reviewed in recent works [54], [55], [56], [57], [58], [59].”