Testing Intrusion detection systems

By John McHugh. ACM Transactions on Privacy and Security, 2000

1,487 citations recorded by the source corpus.

In 1998 and again in 1999, the Lincoln Laboratory of MIT conducted a comparative evaluation of intrusion detection systems (IDSs) developed under DARPA funding. While this evaluation represents a significant and monumental undertaking, there are a number of issues associated with its design and execution that remain unsettled. Some methodologies used in the evaluation are questionable and may have biased its results. One problem is that the evaluators have published relatively little concerning some of the more critical aspects of their work, such as validation of their test data. The appropriateness of the evaluation techniques used needs further investigation. The purpose of this article is to attempt to identify the shortcomings of the Lincoln Lab effort in the hope that future efforts of this kind will be placed on a sounder footing. Some of the problems that the article points out might well be resolved if the evaluators were to publish a detailed description of their procedures and the rationale that led to their adoption, but other problems would clearly remain./par>

Subfields: Artificial Intelligence, Computer Networks and Communications, Information Systems

View Paper: https://www.semanticscholar.org/paper/2c9b78203f61d30cf6c3b04a979554d3e59a3f61

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Information Systems

Citations in the corpus

A Survey of Network-based Intrusion Detection Data Sets (2019, neutral): “the large amount of redundancy [63], [75].”
Unsupervised learning techniques for an intrusion detection system (2004, background): “In fact, in [25] there is a detailed analysis of the shortcomings of this traffic sample set.”
Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization (2018, methodology): “Moreover, it lacks actual attack data records (McHugh, 2000) (Brown et al.”
AdaBoost-Based Algorithm for Network Intrusion Detection (2008, background): “Despite some drawbacks mentioned in [18], it has served as a reliable benchmark data set for many network-based intrusion detection algorithms.”
A survey of intrusion detection in Internet of Things (2017, methodology): “While several works criticize and point out that this is a very outdated dataset, unable to accommodate the latest trend in attacks (McHugh, 2000; Brown et al., 2009; Nehinbe, 2011), having an evaluation dataset is crucial to learn about the correctness of a model.”
Clustering intrusion detection alarms to support root cause analysis (2003, background): “Given that this latter problem is very difficult [McHugh 2000], it seems unlikely that there is a simple way to quantitatively measure the risk that filtering discards true positives.”
Event labeling combining ensemble detectors and background knowledge (2014, neutral): “[37,50]).”
UNSW-NB15: a comprehensive data set for network intrusion detection systems (UNSW-NB15 network data set) (2015, background): “Second, there are also multiple missing records that are a factor in changing the nature of the data [9].”
Ethical Dilemmas in Take-Down Research (2011, background): “Fabricating plausible incidents may be hard, and there is a history of difficulty in other areas of information security that have tried to rely on ‘synthetic’ data, notably in intrusion detection [12].”
Random-Forests-Based Network Intrusion Detection Systems (2008, background): “from the limitations of the Defence Advanced Research Projects Agency (DARPA) datasets [38].”
Anomaly-based intrusion detection system through feature selection analysis and building hybrid efficient model (2017, background): “However, [8] demonstrated how DARPA datasets are not suited for simulating actual network systems.”
A Review of Intrusion Detection Systems Using Machine and Deep Learning in Internet of Things: Challenges, Solutions and Future Directions (2020, background): “However, analysis suggests that the KDD99 dataset negatively affects the IDS results in [199,200] and [201].”
Comprehensive approach to intrusion detection alert correlation (2004, background): “The DARPA sponsored data sets are all synthetically generated, and questions have been raised about the realism of the background traffic models used [29].”
IP covert timing channels: design and detection (2004, background): “Indeed, studies have shown that because the normal traffic in the DARPA dataset was synthetically generated, it is not entirely representative of real traffic [24, 22].”