Stateful Detection of Black-Box Adversarial Attacks
By Steven Chen, Nicholas Carlini, David A. Wagner. 2019
148 citations recorded by the source corpus.
The problem of adversarial examples, evasion attacks on machine learning classifiers, has proven extremely difficult to solve. This is true even in the black-box threat model, as is the case in many practical settings. Here, the classifier is hosted as a remote service and the adversary does not have direct access to the model parameters. This paper argues that in such settings, defenders have a larger space of actions than previously studied. Specifically, we deviate from the implicit assumption made by prior work that a defense must be a stateless function that operates on individual examples, and evaluate the space of stateful defenses. We develop a defense designed to detect the process of generating adversarial examples. By keeping a history of the past queries, a defender can try to identify when a sequence of queries appears to be for the purpose of generating an adversarial example. We then introduce query blinding, a new class of attacks designed to bypass defenses that rely on such a defense approach. We believe that expanding the study of adversarial examples from stateless classifiers to stateful systems is not only more realistic for many black-box settings, but also gives the defender a much-needed advantage in responding to the adversary.
Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Nicholas Carlini
- Dawn Song
- Jacob Steinhardt
- Florian Tramèr
- Nicolas Papernot
- Percy Liang
- Dan Hendrycks
- Matthias Hein
In Safety, Robustness & Interpretability
- Nicholas Carlini
- Jacob Steinhardt
- Florian Tramèr
- Dan Hendrycks
- Nicolas Papernot
- Dawn Song
- Matthias Hein
- Dan Boneh
In Signal Processing
- Florian Tramèr
- Dan Hendrycks
- Jacob Steinhardt
- Nicholas Carlini
- Nicolas Papernot
- Dan Boneh
- Giancarlo Pellegrino
- Dawn Song
Citations in the corpus
- Unsolved Problems in ML Safety (2021, background): “If a blackbox ML system is not publicly released and can only be queried, it may be possible to practically defend the system against zero-query attacks [189] or limited-query attacks [34].”
- AdVersarial: Perceptual Ad Blocking meets Adversarial Machine Learning (2018, background): “, black-box attacks [19] or physicallyrealizable attacks [21].”
- Sponge Examples: Energy-Latency Attacks on Neural Networks (2020, background): “Depending on the setup both defence [77, 78] and detection [79, 80] mechanisms may be required.”