Summoning Demons: The Pursuit of Exploitable Bugs in Machine Learning
By Rock Stevens, Octavian Suciu, Andrew Ruef, Sanghyun Hong, Michael W. Hicks, Tudor Dumitras. 2017
20 citations recorded by the source corpus.
Governments and businesses increasingly rely on data analytics and machine learning (ML) for improving their competitive edge in areas such as consumer satisfaction, threat intelligence, decision making, and product efficiency. However, by cleverly corrupting a subset of data used as input to a target's ML algorithms, an adversary can perturb outcomes and compromise the effectiveness of ML technology. While prior work in the field of adversarial machine learning has studied the impact of input manipulation on correct ML algorithms, we consider the exploitation of bugs in ML implementations. In this paper, we characterize the attack surface of ML programs, and we show that malicious inputs exploiting implementation bugs enable strictly more powerful attacks than the classic adversarial machine learning techniques. We propose a semi-automated technique, called steered fuzzing, for exploring this attack surface and for discovering exploitable bugs in machine learning programs, in order to demonstrate the magnitude of this threat. As a result of our work, we responsibly disclosed five vulnerabilities, established three new CVE-IDs, and illuminated a common insecure practice across many machine learning systems. Finally, we outline several research directions for further understanding and mitigating this threat.
Subfields: Artificial Intelligence, Software, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Michael Backes
- Weilin Xu
- Dietrich Klakow
- Thomas Alexander Trost
- Kathrin Grosse
- Marius Mosbach
- Matthias Hein
- Nicholas Carlini
In Software
In Signal Processing
- Michael Backes
- Weilin Xu
- Dietrich Klakow
- Kathrin Grosse
- Nicolas Papernot
- Patrick Drew McDaniel
- Nicholas Carlini
Citations in the corpus
- Security Risks in Deep Learning Implementations (2017, background): “While doing this work, we found another group of researchers [17] that have also studied the vulnerabilities and tier impacts on machine learning applications.”
- Attacks on Machine Learning: Lurking Danger for Accountability (2019, unknown): “A -R Classifier Evasion [11] X [7, 48] [20, 48, 55, 68] X”
- Adversarial Initialization - when your network performs the way I want (2019, background): “This includes security relevant bugs in ML frameworks [35], as opposed to our active attacker.”