Prediction Poisoning: Towards Defenses Against DNN Model Stealing Attacks

By Tribhuvanesh Orekondy, Bernt Schiele, Mario Fritz. 2019

200 citations recorded by the source corpus.

High-performance Deep Neural Networks (DNNs) are increasingly deployed in many real-world applications e.g., cloud prediction APIs. Recent advances in model functionality stealing attacks via black-box access (i.e., inputs in, predictions out) threaten the business model of such applications, which require a lot of time, money, and effort to develop. Existing defenses take a passive role against stealing attacks, such as by truncating predicted information. We find such passive defenses ineffective against DNN stealing attacks. In this paper, we propose the first defense which actively perturbs predictions targeted at poisoning the training objective of the attacker. We find our defense effective across a wide range of challenging datasets and DNN model stealing attacks, and additionally outperforms existing defenses. Our defense is the first that can withstand highly accurate model stealing attacks for tens of thousands of queries, amplifying the attacker's error rate up to a factor of 85$\times$ with minimal impact on the utility for benign users.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

The False Promise of Imitating Proprietary LLMs (2023, background): “…this direction, e.g., one can detect whether a particular model is trained via imitation (Krishna et al., 2020; Juuti et al., 2019; Szyller et al., 2019; Maini et al., 2021) or slow model stealing by sacrifing some performance (Wallace et al., 2020; Orekondy et al., 2020; Dziedzic et al., 2022a,b).”
On the Difficulty of Defending Self-Supervised Learning against Model Extraction (2022, methodology): “We can consider the gradient of the loss with respect to the model parameters θ ∈ R D for a query x made by the attacker to formalize this objective as in (Orekondy et al., 2020).”
Advances in adversarial attacks and defenses in computer vision: A survey (2021, background): “A prediction poisoning attack is adopted as a defense against the model stealing attacks in [443].”
Increasing the Cost of Model Extraction with Calibrated Proof of Work (2022, background): “Current active defenses perturb the outputs to poison the training objective of an attacker (Orekondy et al., 2020).”
A Systematic Review on Model Watermarking for Neural Networks (2020, background): “by only returning hard labels and no probabilities per output class, by perturbing the prediction outputs [Orekondy et al., 2019], or by designing networks that are extremely sensitive to weight changes, which makes it difficult for an attacker to steal and adapt them [Szentannai et al.”
Imitation Attacks and Defenses for Black-box Machine Translation Systems (2020, methodology): “To accomplish this, we repurpose prediction poisoning (Orekondy et al., 2020) for MT: rather than outputting the original translation y , we output a different (high-accuracy) translation ˜y that steers the training of the imitation model in the wrong direction.”
A Survey of Privacy Attacks in Machine Learning (2020, background): “Orekondy et al. [99] proposed to perturb the prediction vectors of the target model in such a way that they poison the substitute model created by the attacker.”