Toward Generating a New Intrusion Detection Dataset and Intrusion Traffic Characterization

By Iman Sharafaldin, Arash Habibi Lashkari, Ali Akbar Ghorbani. International Conference on Information Systems Security and Privacy, 2018

4,845 citations recorded by the source corpus.

: With exponential growth in the size of computer networks and developed applications, the signiﬁcant in-creasing of the potential damage that can be caused by launching attacks is becoming obvious. Meanwhile, Intrusion Detection Systems (IDSs) and Intrusion Prevention Systems (IPSs) are one of the most important defense tools against the sophisticated and ever-growing network attacks. Due to the lack of adequate dataset, anomaly-based approaches in intrusion detection systems are suffering from accurate deployment, analysis and evaluation. There exist a number of such datasets such as DARPA98, KDD99, ISC2012, and ADFA13 that have been used by the researchers to evaluate the performance of their proposed intrusion detection and intrusion prevention approaches. Based on our study over eleven available datasets since 1998, many such datasets are out of date and unreliable to use. Some of these datasets suffer from lack of trafﬁc diversity and volumes, some of them do not cover the variety of attacks, while others anonymized packet information and payload which cannot reﬂect the current trends, or they lack feature set and metadata. This paper produces a reliable dataset that contains benign and seven common attack network ﬂows, which meets real world criteria and is publicly avaliable. Consequently, the paper evaluates the performance of a comprehensive set of network trafﬁc features and machine learning algorithms to indicate the best set of features for detecting the certain attack categories.

Subfields: Computer Networks and Communications

View Paper: https://www.semanticscholar.org/paper/a27089efabc5f4abd5ddf2be2a409bff41f31199

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

Citations in the corpus

TON_IoT Telemetry Dataset: A New Generation Dataset of IoT and IIoT for Data-Driven Intrusion Detection Systems (2020, methodology): “Various network datasets, for example, KDDCUP99, NSL-KDD [12], UNSW-NB15 [13] and ISCX [14], were generated for evaluating IDSs; however, they do not include any speciﬁc characteristics of IoT/IIoT applications as these datasets contain neither sensors’ reading data (i.e., telemetry or measurement d”
Deep Learning for Time Series Anomaly Detection: A Survey (2022, neutral): “CalIt2 [57] Real MTS 10,080 2 2 Urban events management CAP [168] [69] Real MTS 921,700,000 108 21 Medical and health CICIDS2017 [155] Real MTS 2,830,540 15 83 Server machines monitoring Credit Card fraud detection [47] Real MTS 284,807 1 31 Fraud detectcion DMDS [3] Real MTS 725,402 1 32 Industrial”
NetFlow Datasets for Machine Learning-based Network Intrusion Detection Systems (2020, methodology): “– CSE-CIC-IDS2018- A dataset released by a collaborative project between the Communications Security Establishment (CSE) & Canadian Institute for Cybersecurity (CIC) in 2018 [14].”
A Novel PCA-Firefly Based XGBoost Classification Model for Intrusion Detection in Networks Using GPU (2020, background): “Moreover, there are cases of anonymous packet information, insufficient payload information, lack of features and metadata [13].”
Dynamically Weighted Balanced Loss: Class Imbalanced Learning and Confidence Calibration of Deep Neural Networks (2021, methodology): “1) Dataset Description: We rely on an intrusion detection dataset (CICIDS2017) published by the Canadian Institute for Cyber-Security (CIC) at the University of New Brunswick (UNB) [54].”
ET-BERT: A Contextualized Datagram Representation with Pre-training Transformers for Encrypted Traffic Classification (2022, methodology): “This dataset contains two parts: (1) about 15GB traffic from the public datasets [9, 30]; (2) about 15GB traffic from our passively collected traffic under the China Science and Technology Network (CSTNET).”
Network Intrusion Detection for IoT Security Based on Learning Techniques (2019, methodology): “Profile [94] approach to outline the behavior on HTTP, HTTPS, FTP, SSH and e-mail protocols.”
Towards a Standard Feature Set for Network Intrusion Detection System Datasets (2021, methodology): “Four widely used NIDS datasets, referred to as UNSW-NB15 [10], BoT-IoT [7], ToN-IoT [1], and CICCSE-IDS2018 [16] have been converted into a common basic NetFlow-based feature set [15].”
A Survey on Machine Learning Techniques for Cyber Security in the Last Decade (2020, methodology): “The bot-IoT dataset was proposed in 2018 for IoT devices [175].”
Deep Learning Models for Cyber Security in IoT Networks (2019, methodology): “For conducting proposed work we have used latest DDoS attack CICIDS2017 dataset[17, 18].”
Towards the Development of Realistic Botnet Dataset in the Internet of Things for Network Forensic Analytics: Bot-IoT Dataset (2018, methodology): “Comparable to [25, 26] but more elaborate in their implementation, Sharafaldin et al. [8] crafted a testbed relying on physical machines, which were separated into two networks, the Victim-network and the Attack-network.”
A Review of Intrusion Detection Systems Using Machine and Deep Learning in Internet of Things: Challenges, Solutions and Future Directions (2020, methodology): “This organization has three different datasets: (1) the CAIDA OC48, covering various types of data observed on an OC48 link, (2) the CAIDA DDOS, which comprises of one-hour DDoS attack traffic that happened in August 2007, and (3) the CAIDA Internet traces 2016, which is passive traffic traces from ”
A Review on Machine Learning and Deep Learning Perspectives of IDS for IoT: Recent Updates, Security Issues, and Challenges (2020, methodology): “Though this dataset has been widely accepted for evaluating IDS, still its precision and absence of real-world attack scenarios were criticized in [186–188].”