On-line outlier detection and data cleaning

By Hancong Liu, Sirish L. Shah, Wei Jiang. Computers and Chemical Engineering, 2004

503 citations recorded by the source corpus.

Outliers are observations that do not follow the statistical distribution of the bulk of the data, and consequently may lead to erroneous results with respect to statistical analysis. Many conventional outlier detection tools are based on the assumption that the data is identically and independently distributed. In this paper, an outlier-resistant data filter-cleaner is proposed. The proposed data filter-cleaner includes an on-li ne outlier-resistant estimate of the process model and combines it with a modified Kalman filter to detect and “clean” outliers. The advantage over existing methods is that the proposed method has the following features: (a) a priori knowledge of the process model is not required; (b) it is applicable to autocorrelated data; (c) it can be implemented on-line; and (d) it tries to only clean (i.e., detects and replaces) outliers and preserves all other information in the data. © 2004 Elsevier Ltd. All rights reserved.

Subfields: Artificial Intelligence, Statistics and Probability

View Paper: https://www.semanticscholar.org/paper/65220a315251eceacd9022e124c8c7b84b58e4e9

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Generalized Out-of-Distribution Detection: A Survey (2021, unknown): “Recent works attempt several less principled approximations including MC-dropout (Gal & Ghahramani, 2016) and deep ensembles (Dietterich, 2000; Lakshminarayanan et al., 2017; Maddox et al., 2019) for faster and better estimates of uncertainty.”
Review of soft sensor methods for regression applications (2016, background): “liers, and it has shown to be practically effective in real applications [21; 19].”
A comparative evaluation of outlier detection algorithms: Experiments and analyses (2018, methodology): “An outlier-free dataset allows for accurate modelling tasks, making outlier detection methods extremely valuable for data cleaning [18].”
Data Mining and Analytics in the Process Industry: The Role of Machine Learning (2017, background): “be removed from the modeling dataset, which will otherwise greatly deteriorate the performance of the machine learning model [19].”
Robust Actor-Critic Contextual Bandit for Mobile Health (mHealth) Interventions (2018, methodology): “We improve the above three methods by first using the state-of-the-art outlier filter [11, 23] to remove outliers, then employing the above three contextual bandit algorithms for the SDM task.”