SoK: Security and Privacy in Machine Learning

By Nicolas Papernot, Patrick Drew McDaniel, Arunesh Sinha, Michael P. Wellman. 2018

626 citations recorded by the source corpus.

Advances in machine learning (ML) in recent years have enabled a dizzying array of applications such as data analytics, autonomous systems, and security diagnostics. ML is now pervasive—new systems and models are being deployed in every domain imaginable, leading to widespread deployment of software based inference and decision making. There is growing recognition that ML exposes new vulnerabilities in software systems, yet the technical community's understanding of the nature and extent of these vulnerabilities remains limited. We systematize findings on ML security and privacy, focusing on attacks identified on these systems and defenses crafted to date.We articulate a comprehensive threat model for ML, and categorize attacks and defenses within an adversarial framework. Key insights resulting from works both in the ML and security communities are identified and the effectiveness of approaches are related to structural elements of ML algorithms and the data used to train them. In particular, it is apparent that constructing a theoretical understanding of the sensitivity of modern ML algorithms to the data they analyze, à la PAC theory, will foster a science of security and privacy in ML.

Subfields: Artificial Intelligence, Computer Networks and Communications, Safety, Robustness & Interpretability, Privacy & Security, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Safety, Robustness & Interpretability

In Privacy & Security

In Signal Processing

Citations in the corpus

Certified Defenses for Data Poisoning Attacks (2017, background): “…striking recent security vulnerability discovered in machine learning systems is adversarial test images that can fool image classiﬁers despite being imperceptible from normal images (Szegedy et al., 2014; Goodfellow et al., 2015; Carlini et al., 2016; Kurakin et al., 2016; Papernot et al., 2016a).”
STRIP: a defence against trojan attacks on deep neural networks (2019, background): “Previous work on poisoning attacks usually aim to degrade a classifier’s accuracy to clean inputs [24], [25].”
Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks (2017, methodology): “Papernot et al. [20] provide a comprehensive summary of recent work on defending against adversarial samples, grouping work into two broad categories: adversarial training and gradient masking , which we discuss further below.”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, methodology): “[26] provided a comprehensive overview of security issues in ML and recent findings in DL and established a unifying threat model.”
An Overview of Federated Deep Learning Privacy Attacks and Defensive Strategies (2020, background): “ssiﬁcation Here we discuss only adversarial example attacks as a form of misclassiﬁcation attacks. Adversarial example attacks aim to produce samples that will be purposefully misclassiﬁed at runtime [31], [32]. Federated learning is vulnerable to this because the deployment of the model means that ”
A Survey on Security Threats and Defensive Techniques of Machine Learning: A Data Driven View (2018, background): “[16] reviewed existing works about security issues and corresponding defensive methods in the life cycle of a machine learning-based system from training to inference.”
Attacks on Machine Learning: Lurking Danger for Accountability (2019, background): “Adversarial Label Flips [76] X X [56] X”
Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defenses (2020, background): “A broad discussion concerning security for machine learning can be found in [10] and [11].”
Detecting AI Trojans Using Meta Neural Analysis (2019, background): “However, most conventional poisoning attacks seek to degrade a model’s classiﬁcation accuracy on clean inputs [7], [44].”
A Survey of Privacy Attacks in Machine Learning (2020, background): “The security of machine learning and the impacts of adversarial attacks in the performance of the models have been widely studied in the community, with several surveys highlighting the major advances in the area [6, 57, 75, 98].”