Cost-based modeling for fraud and intrusion detection: results from the JAM project

By Salvatore J. Stolfo, Wei Fan, Wenke Lee, Andreas L. Prodromidis, Philip Ka-Fai Chan. Proceedings DARPA Information Survivability Conference and Exposition. DISCEX'00, 2000

703 citations recorded by the source corpus.

We describe the results achieved using the JAM distributed data mining system for the real world problem of fraud detection in financial information systems. For this domain we provide clear evidence that state-of-the-art commercial fraud detection systems can be substantially improved in stopping losses due to fraud by combining multiple models of fraudulent transaction shared among banks. We demonstrate that the traditional statistical metrics used to train and evaluate the performance of learning systems (ie. statistical accuracy or ROC analysis) are misleading and perhaps inappropriate for this application. Cost-based metrics are more relevant in certain domains, and defining such metrics poses significant and interesting research questions both in evaluating systems and alternative models, and in formalizing the problems to which one may wish to apply data mining technologies. This paper also demonstrates how the techniques developed for fraud detection can be generalized and applied to the important area of intrusion detection in networked information systems. We report the outcome of recent evaluations of our system applied to tcpdump network intrusion data specifically with respect to statistical accuracy. This work involved building additional components of JAM that we have come to call, MADAM ID (Mining Audit Data for Automated Models for Intrusion Detection). However, taking the next step to define cost-based models for intrusion detection poses interesting new research questions. We describe our initial ideas about how to evaluate intrusion detection systems using cost models learned during our work on fraud detection.

Subfields: Artificial Intelligence, Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/9eb3f14a80576a8aad6f3e9f2a660da9d512a962

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Information Systems

Citations in the corpus

Credit Card Fraud Detection Using Hidden Markov Model (2012, methodology): “For comparison, we consider the metrics TP and FP, as well as TP-FP and Accuracy [7].”
Outlier Analysis (2013, background): “A wide variety of methods [35, 106, 184, 367, 425, 426] are available for credit-card fraud detection, though the general experience has been that supervised methods are the most effective.”
Network Anomaly Detection: Methods, Systems and Tools (2014, methodology): “[206] and is built on the data captured in the DARPA98 IDS evaluation program.”
Anomaly-based network intrusion detection: Techniques, systems and challenges (2009, background): “The absence of appropriate metrics and assessment meth- odologies, as well as a general framework for evaluating and comparing alternative IDS techniques (Stolfo and Fan, 2000; Gaffney and Ulvila, 2001).”
Hidden Markov Model (2010, positive): “For comparison, we consider the metrics TP and FP, as well as TP-FP and Accuracy [7].”
Adaptive Anomaly Detection via Self-calibration and Dynamic Updating (2009, background): “JAM [27] focuses on developing and evaluating a range of learning strategies for fraud detection.”
Casting out Demons: Sanitizing Training Data for Anomaly Sensors (2008, methodology): “The exchange of abnormal models was used in [24] for commercial fraud detection.”
A Deep Learning Approach to Network Intrusion Detection (2018, methodology): “The KDD Cup ’99 dataset was used in DARPA’s IDS evaluation program [34].”
Statistical fraud detection: A review (2002, methodology): “Combinations of some or all of these algorithms can be created using meta-learning algorithms to improve prediction in fraud detection (e.g., Chan, Fan, Prodromidis and Stolfo, 1999).”
A framework for the evaluation of intrusion detection systems (2006, positive): “In this section we present the expected cost of an IDS by combining some of the ideas originally presented in [9] and [28].”
Intrusion detection using an ensemble of intelligent paradigms (2005, background): “Several researchers proposed data mining techniques to identify key patterns that help in detecting intrusions (Stolfo et al., 2000; Jianxiong and Bridges, 2000; Mukkamala and Sung, 2003).”
A detailed analysis of the KDD CUP 99 data set (2009, methodology): “This data set is prepared by Stolfo et al. [5] and is built based on the data captured in DARPA’98 IDS evaluation program [6].”
A Survey of Clustering Algorithms for Big Data: Taxonomy and Empirical Analysis (2014, methodology): “We experimented also with two other publicly available datasets, namely DARPA [35] and internet traffic data (ITD) [29].”
An overview of anomaly detection techniques: Existing solutions and latest technological trends (2007, methodology): “Although the Receiver Operating Characteristic (ROC) curve has been widely used to evaluate the accuracy of intrusion detection systems and analyze the tradeoff between the false positives rate and the detection rate, evaluations based on the ROC curve are often misleading and/or incomplete [91,92].”
Building an Intrusion Detection System Using a Filter-Based Feature Selection Algorithm (2016, methodology): “The data obtained during the phase of data collection are first processed to generate the basic features such as the ones in KDD Cup 99 dataset [41].”
DeepDefense: Identifying DDoS Attack via Deep Learning (2017, background): “Other available public datasets are even older: KDD Cup 1999 [38], MIT Lincoln Laboratory DARPA 2000 [39], CAIDA DDoS Attack 2007 [40], and TUIDS DDoS dataset 2012 [41].”