Hacking smart machines with smarter ones: How to extract meaningful data from machine learning classifiers

By Giuseppe Ateniese, Luigi Vincenzo Mancini, Angelo Spognardi, Antonio Villani, Domenico Vitali, Giovanni Felici. 2013

494 citations recorded by the source corpus.

Machine Learning (ML) algorithms are used to train computers to perform a variety of complex tasks and improve with experience. Computers learn how to recognize patterns, make unintended decisions, or react to a dynamic environment. Certain trained machines may be more effective than others because they are based on more suitable ML algorithms or because they were trained through superior training sets. Although ML algorithms are known and publicly released, training sets may not be reasonably ascertainable and, indeed, may be guarded as trade secrets. While much research has been performed about the privacy of the elements of training sets, in this paper we focus our attention on ML classifiers and on the statistical information that can be unconsciously or maliciously revealed from them. We show that it is possible to infer unexpected but useful information from ML classifiers. In particular, we build a novel meta-classifier and train it to hack other classifiers, obtaining meaningful information about their training sets. This kind of information leakage can be exploited, for example, by a vendor to build more effective classifiers or to simply acquire trade secrets from a competitor's apparatus, potentially violating its intellectual property rights.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Red Teaming Language Models with Language Models (2022, background): “…leakage can be used to infer the data used in training (“membership inference”; Shokri et al., 2017; Song and Shmatikov, 2019; Nasr et al., 2019; Hisamoto et al., 2020; Carlini et al., 2021), helping adversaries clone private, commercial LMs, violating intellectual property (Ateniese et al., 2013).”
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks (2018, background): “[2] show that if an adversary is given access to a remote machine learning model (e.”
Towards the Science of Security and Privacy in Machine Learning (2016, background): “infer statistical information about the training dataset from a trained model hθ [55]: i.”
Membership Inference Attacks Against Machine Learning Models (2016, methodology): “In [2], knowledge of the parameters of SVM and HMM models is used to infer general statistical information about the training dataset, for example, whether records of a particular race were used during training.”
Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, background): “Specifically, an attacker could compromise the confidentiality of users’ private training or testing data via various attacks such as model inversion attacks [21, 22], membership inference attacks [40, 49, 53], and property inference attacks [1, 24].”
Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, background): “It is relatively straightforward to extract sensitive information from a model [4, 26, 27].”
Privacy in Deep Learning: A Survey (2020, background): “Inference Parameter Inference Property Inference Access to Model Access to Model Output Membership Inference [13] # # # # Blackbox Logits Measuring Membership Privacy[35] # # # # Blackbox Logits ML-Leaks[36] # # # # Blackbox Logits The Natural Auditor [37] # # # # Blackbox Label LOGAN [38] # # # # B”
Stealing Machine Learning Models via Prediction APIs (2016, background): “An adversary may use knowledge of the ML model to avoid detection by it [4, 9, 29, 36, 55].”
Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks (2018, background): “[16] showed if an adversary has access to the parameters of machine learning models such as Support Vector Machines (SVM) or Hidden Markov Models (HMM) [17], she can extract valuable information about the training data (e.”
Machine Learning Models that Remember Too Much (2017, background): “[4] show how an attacker can use access to an ML model to infer a predicate of the training data, e.”
Attacking Graph-based Classification via Manipulating the Graph Structure (2019, background): “…examples [7,12,27,30,37,40,43,46,51,53,56, 68], poisoning attacks [8,18,31,38,67,70], privacy attacks for users (e.g., model inversion attacks [19,20], membership inference attacks [49,52], property inference attacks [3,21]), as well as model parameter and hyperparameter stealing attacks [60,61].”
Property Inference Attacks on Fully Connected Neural Networks using Permutation Invariant Representations (2018, methodology): “[5] used the emission and transition probabilities of each node in an HMM separately as a data sample for the meta-classifier.”
Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning (2018, background): “[36] show that an adversary with access to the parameters of machine learning models such as Support Vector Machines (SVM) or Hidden Markov Models (HMM) [37] can extract valuable information about the training data (e.”
LOGAN: Membership Inference Attacks Against Generative Models (2017, background): “That is, the adversary ﬁrst infers whether data of a victim is part of the information she has access to (a trained model in our case), and then mount other attacks (e.g., proﬁling [49], property inference [4, 41], etc.), which might leak additional information about the victim.”
SoK: Security and Privacy in Machine Learning (2018, background): “White-Box Full No [42], [43], [44] [24], [21], [44], [45], [46] [47] [48]”
Measuring Forgetting of Memorized Training Examples (2022, background): “While other attacks exist, such as attribute inference [23] or property inference [3], these attacks tend to capture properties of many examples or even the entire training set, whereas membership inference and training data extraction are designed to measure the privacy of few or a single example.”