Towards the Science of Security and Privacy in Machine Learning

By Nicolas Papernot, Patrick Drew McDaniel, Arunesh Sinha, Michael P. Wellman. 2016

507 citations recorded by the source corpus.

Advances in machine learning (ML) in recent years have enabled a dizzying array of applications such as data analytics, autonomous systems, and security diagnostics. ML is now pervasive---new systems and models are being deployed in every domain imaginable, leading to rapid and widespread deployment of software based inference and decision making. There is growing recognition that ML exposes new vulnerabilities in software systems, yet the technical community's understanding of the nature and extent of these vulnerabilities remains limited. We systematize recent findings on ML security and privacy, focusing on attacks identified on these systems and defenses crafted to date. We articulate a comprehensive threat model for ML, and categorize attacks and defenses within an adversarial framework. Key insights resulting from works both in the ML and security communities are identified and the effectiveness of approaches are related to structural elements of ML algorithms and the data used to train them. We conclude by formally exploring the opposing relationship between model accuracy and resilience to adversarial manipulation. Through these explorations, we show that there are (possibly unavoidable) tensions between model complexity, accuracy, and resilience that must be calibrated for the environments in which they will be used.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Certified Robustness to Adversarial Examples with Differential Privacy (2018, background): “dients phenomenon, also referred to as gradient masking [45], [55], makes the defense susceptible to new attacks crafted to circumvent that obfuscation [2].”
Detecting Backdoor Attacks on Deep Neural Networks by Activation Clustering (2018, methodology): “Prior work (Barreno et al., 2010; Nelson, 2010; Huang et al., 2011; Papernot et al., 2016) has shown that this type of attack, called a poisoning attack, can lead to poor model performance, models that are easily fooled, and targeted misclassifications, exposing safety risks.”
Certified Defenses for Data Poisoning Attacks (2017, background): “…striking recent security vulnerability discovered in machine learning systems is adversarial test images that can fool image classiﬁers despite being imperceptible from normal images (Szegedy et al., 2014; Goodfellow et al., 2015; Carlini et al., 2016; Kurakin et al., 2016; Papernot et al., 2016a).”
Ensemble Adversarial Training: Attacks and Defenses (2017, background): “Given the ubiquity of this pitfall among proposed defenses against adversarial examples [4, 3, 15], we advise researchers to always consider both white-box and black-box adversaries when evaluating defensive strategies.”
Machine Learning Models that Remember Too Much (2017, background): “ by a ML model. They were first explored in the context of spam detection [28, 50, 51]. More recent work investigated evasion in other settings such as computer vision—see a survey by Papernot et al. [63]. Our work focuses on the confidentiality of training data rather than evasion, but future work ”
Stealing Hyperparameters in Machine Learning (2018, background): “In particular, the attacker could insert new instances, edit existing instances, or remove existing instances in the training dataset [38].”
Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks (2017, methodology): “concluded that methods designed to conceal gradient information are bound to have limited success because of the transferability of adversarial examples [35].”
On the (Statistical) Detection of Adversarial Examples (2017, background): “We also derive a formal argument justifying the statistical divergence of adversarial examples from benign training points.”
Defense-GAN: Protecting Classifiers Against Adversarial Attacks Using Generative Models (2018, methodology): “Defensive distillation (Papernot et al., 2016d) trains the classiﬁer in two rounds using a variant of the distillation (Hinton et al., 2014) method.”
Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning (2018, background): “c) Robustness: Robustness to input perturbation is another important requirement for security [26], [27], [28], [29], [30] and safety [31] in ML systems.”
Adversarial Examples: Attacks and Defenses for Deep Learning (2017, methodology): “[37] covered a wide range of security problems.”
Adversarially Robust Generalization Requires More Data (2018, background): “[43] discuss adversarial robustness at the population level.”
The Space of Transferable Adversarial Examples (2017, background): “Adversarial examples are maliciously perturbed inputs designed to mislead machine learning (ML) models at test-time.”
Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey (2018, background): “self driving cars [12], [180], surveillance [13], maleware detection [34], [107], drones and robotics [155], [178], and voice command recognition [7].”
Adversarial Attacks on Neural Network Policies (2017, background): “Adversarial machine learning [2], and more generally the security and privacy of machine learning [18], encompasses a line of work that seeks to understand the behavior of models and learning algorithms in the presence of adversaries.”
Adversarial Attacks and Defences: A Survey (2018, background): “We begin with the identification of threat surface [61] of systems built on machine learning models to identify where and how an adversary may attempt to subvert the system under attack.”