Stealing Machine Learning Models via Prediction APIs
By Florian Tramèr, Fan Zhang, Ari Juels, Michael K. Reiter, Thomas Ristenpart. 2016
2,163 citations recorded by the source corpus.
Machine learning (ML) models may be deemed confidential due to their sensitive training data, commercial value, or use in security applications. Increasingly often, confidential ML models are being deployed with publicly accessible query interfaces. ML-as-a-service ("predictive analytics") systems are an example: Some allow users to train models on potentially sensitive data and charge others for access on a pay-per-query basis. The tension between model confidentiality and public access motivates our investigation of model extraction attacks. In such attacks, an adversary with black-box access, but no prior knowledge of an ML model's parameters or training data, aims to duplicate the functionality of (i.e., "steal") the model. Unlike in classical learning theory settings, ML-as-a-service offerings may accept partial feature vectors as inputs and include confidence values with predictions. Given these practices, we show simple, efficient attacks that extract target ML models with near-perfect fidelity for popular model classes including logistic regression, neural networks, and decision trees. We demonstrate these attacks against the online services of BigML and Amazon Machine Learning. We further show that the natural countermeasure of omitting confidence values from model outputs still admits potentially harmful model extraction attacks. Our results highlight the need for careful ML model deployment and new model extraction countermeasures.
Subfields: Artificial Intelligence, Computer Networks and Communications, Safety, Robustness & Interpretability, Privacy & Security
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Bernt Schiele
- Mohit Bansal
- Yoshua Bengio
- Nicholas Carlini
- Quoc V. Le
- Nicolas Papernot
- Dawn Song
In Computer Networks and Communications
- Neil Zhenqiang Gong
- Binghui Wang
- Battista Biggio
- Giorgio Giacinto
- Prateek Mittal
- Jinyuan Jia
- Nicholas Carlini
- Nicolas Papernot
In Safety, Robustness & Interpretability
- Nicolas Papernot
- Nicholas Carlini
- Dawn Song
- Bo Li
- Florian Tramèr
- Matthias Hein
- Bernt Schiele
- Prateek Mittal
In Privacy & Security
- Peter Kairouz
- Vitaly Shmatikov
- Dan Boneh
- Reza Shokri
- Rasmus Pagh
- H. Brendan McMahan
- Abhradeep Thakurta
- Badih Ghazi
Citations in the corpus
- The False Promise of Imitating Proprietary LLMs (2023, methodology): “Past work on model imitation Prior work has shown that model imitation is possible for various domains (Orekondy et al., 2019; Tram ` er et al., 2016; Lowd and Meek, 2005), including language classifiers (Krishna et al., 2020; Pal et al., 2019) and machine translation systems (Wallace et al., 2020).”
- The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks (2018, background): “Model stealing studies a related problem to training data extraction: under a black-box threat model, model stealing attempts to extract the parameters θ (or parameters similar to them) from a remote model, so that the adversary can have their own copy [49].”
- Towards the Science of Security and Privacy in Machine Learning (2016, background): “Tramer et al. show how to extract parameters of a model from the observation of its predictions [60].”
- Membership Inference Attacks Against Machine Learning Models (2016, methodology): “Model extraction attacks [32] aim to extract the parameters of a model trained on private data.”
- Advances and Open Problems in Federated Learning (2019, background): “It is crucial to note that even if the model parameters themselves are successfully hidden, research has shown that in many cases they can be reconstructed by an adversary who only has access to an inference/prediction API based on those parameters [384].”
- Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, background): “Moreover, an attacker could also compromise the confidentiality/intellectual property of a model provider via stealing its model parameters and hyperparameters [34, 58, 60].”
- Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, background): “Recently, the work [81] shows that stealing machine learning models is possible when taking into consideration only the predictions provided by the model.”
- Privacy in Deep Learning: A Survey (2020, background): “Inference Parameter Inference Property Inference Access to Model Access to Model Output Membership Inference [13] # # # # Blackbox Logits Measuring Membership Privacy[35] # # # # Blackbox Logits ML-Leaks[36] # # # # Blackbox Logits The Natural Auditor [37] # # # # Blackbox Label LOGAN [38] # # # # B”
- Secure Evaluation of Quantized Neural Networks (2019, background): “It is nevertheless worth noting that an adversary with black-box access to a model, such as in the Machine Learning As A Service scenario, who can request predictions on arbitrary inputs can steal the model with nearperfect fidelity in some cases [62, 64].”
- Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, background): “Attacks targeting model privacy include: the adversary uncovering the model details with model extraction attacks [58], and inferring hyperparameters with hyperparameter stealing attacks [60].”
- Data Poisoning against Differentially-Private Learners: Attacks and Defenses (2019, methodology): “More subtle methods aim to recover training examples [8], or even extract the model itself [20].”
- Ekiden: A Platform for Confidentiality-Preserving, Trustworthy, and Performant Smart Contracts (2018, neutral): “, a secret prediction model may be “extracted” via client queries [87].”
- Ensemble Adversarial Training: Attacks and Defenses (2017, background): “[19] showed that these inputs transfer (or generalize) across models, thus enabling black-box attacks on deployed models [13, 9]: an adversary can train a local model (possibly by issuing prediction queries to the target model [13, 21]), find an adversarial example on the local model, and apply it t”
- Practical Black-Box Attacks against Machine Learning (2016, methodology): “Our contributions are: • We introduce in Section 4 an attack against black-box DNN classifiers.”
- Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks (2018, background): “Input inference [20], attribute inference [21], parameter inference [22], [23], and side-channel attacks [24] are several examples of such attacks.”