Robust kernel density estimation

By JooSeuk Kim, Clayton D. Scott. 2008

439 citations recorded by the source corpus.

In this paper, we propose a method for robust kernel density estimation. We interpret a KDE with Gaussian kernel as the inner product between a mapped test point and the centroid of mapped training points in kernel feature space. Our robust KDE replaces the centroid with a robust estimate based on M-estimation (P. Huber, 1981), The iteratively re-weighted least squares (IRWLS) algorithm for M-estimation depends only on inner products, and can therefore be implemented using the kernel trick. We prove the IRWLS method monotonically decreases its objective value at every iteration for a broad class of robust loss functions. Our proposed method is applied to synthetic data and network traffic volumes, and the results compare favorably to the standard KDE.

Subfields: Artificial Intelligence, Control and Systems Engineering, Statistics and Probability, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Statistics and Probability

In Probabilistic & Bayesian Methods

Citations in the corpus

Kernel Mean Embedding of Distributions: A Review and Beyonds (2016, background): “As an aside, Kim and Scott (2012) also exploit this interpretation to robustify the kernel density estimator (KDE) by replacing the squared loss L(k(xi, ·), g) = ‖g − k(xi, ·)‖2H in (3.24) by the loss which is less sensitive to outliers such as Huber loss (Huber 1964) and k is assumed to be…”
Kernel Mean Shrinkage Estimators (2014, background): “…g ) b Given these formulations, it is natural to ask if minimizing the regularized version of E ( g ) will give a “better” estimator. respectively (Kim and Scott, 2012). b While this question is interesting, it has to be noted that in principle, there is really no need to consider a regularized…”
Kernel Mean Estimation and Stein's Effect (2013, positive): “In [21], the loss functional Ê(g) is replaced by a robust loss function such as the Huber’s loss to reduce the effect of outliers.”
Comparing Recognition Performance and Robustness of Multimodal Deep Learning Models for Multimodal Emotion Recognition (2021, background): “2) BDAE minimizes the mean squared error, which is sensitive to outliers [55].”
Generative Probabilistic Novelty Detection with Adversarial Autoencoders (2018, methodology): “Novelty detection methods can be statistical and probabilistic based [15, 16], distance based [17], and also based on self-representation [8].”
A comparative evaluation of outlier detection algorithms: Experiments and analyses (2018, methodology): “in [13] where the authors describe a Robust Kernel Density Estimator (RKDE), algorithm which uses Mestimation methods, such as robust loss functions, to overcome the limitations of plain kde.”
Deep Anomaly Detection Using Geometric Transformations (2018, methodology): “Examples of such methods are kernel density estimation (KDE) [22], and Robust-KDE [16].”
MacroBase: Prioritizing Attention in Fast Data (2016, background): “ay not be meaningful. Similarly, the MCD assumes data is drawn from a single, ellipsoidal distribution. Recent proposals for robust non-parametric estimators, such as robust kernel density estimation [32], as well as classic methods for reducing dimensionality, can help alleviate these problems. How”
Deep Semi-Supervised Anomaly Detection (2019, background): “…such as the One-Class SVM (OC-SVM) [60], Support Vector Data Description (SVDD) [65], Isolation Forest (IF) [38], or Kernel Density Estimator (KDE) [50, 31, 68] often require manual feature engineering to be effective on high-dimensional data and are limited in their scalability to large datasets.”
Deep Structured Energy Based Models for Anomaly Detection (2016, methodology): “…outlier detection methods, PCA and Kernel PCA, two density-based methods Kernel Density Estimator (KDE) (Parzen, 1962) and Robust Kernel Density Estimator (RKDE) (Kim & Scott, 2012), along with the traditional one-class learning method One-Class SVM (OC-SVM) (Sch¨olkopf et al., 2001).”
MacroBase: Analytic Monitoring for the Internet of Things (2016, background): “Recent proposals for robust non-parametric estimators, such as robust kernel density estimation [32], as well as classic methods for reducing dimensionality, can help alleviate these problems.”
Statistical Analysis of Nearest Neighbor Methods for Anomaly Detection (2019, methodology): “Many empirical methods have been developed in the unsupervised setup, which can be roughly classified into four categories: density based methods such as the Robust KDE (RKDE) [4], Local Outlier Factor (LOF) [5], and mixture models (EGMM); distance based methods such as kNN [6] and Angle-based Outli”
A Unifying Review of Deep and Shallow Anomaly Detection (2020, methodology): “For shallow methods, the problem is well studied, and we have many effective methods [5], [259], [326], [372], [374], [375], [471].”
Bayesian Optimization for Likelihood-Free Inference of Simulator-Based Statistical Models (2015, background): “…definite, so that the estimated density is a member of a reproducing kernel Hilbert space where more robust approximations of pΦ|θ than the one in Equation (18) would exist (Kim and Scott, 2012), and moreover, one could investigate connections to the inference approach of Fukumizu et al. (2013).”
HRN: A Holistic Approach to One Class Learning (2020, background): “…anomaly detection algorithms can be classiﬁed into the following categories: distance based methods [42, 3, 28, 31, 60], density based methods [38, 56, 6], mixture models [3, 43], one-class classiﬁcation based methods [75, 78, 39], deep learning based representation learning using…”
Unsupervised Deep Anomaly Detection for Multi-Sensor Time-Series Signals (2021, methodology): “Digital Object Identifier no. 10.1109/TKDE.2021.3102110 1041-4347 © 2021 IEEE.”
Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection (2018, methodology): “Clustering analysis is another popular category of methods used for density estimation and anomaly detection, such as multivariate Gaussian Models, Gaussian Mixture Models, k-means, and so on (Barnett & Lewis (1984); Zimek et al. (2012); Kim & Scott (2011); Xiong et al. (2011)).”
Learning Discriminative Reconstructions for Unsupervised Outlier Removal (2015, methodology): “Statistical methods [6, 27, 11] fit parametric distributions on data, and outliers are identified as those having low probability under the learned distributions.”