Property Inference Attacks on Fully Connected Neural Networks using Permutation Invariant Representations

By Karan Ganju, Qi Wang, Wei Yang, Carl A. Gunter, Nikita Borisov. 2018

598 citations recorded by the source corpus.

With the growing adoption of machine learning, sharing of learned models is becoming popular. However, in addition to the prediction properties the model producer aims to share, there is also a risk that the model consumer can infer other properties of the training data the model producer did not intend to share. In this paper, we focus on the inference of global properties of the training data, such as the environment in which the data was produced, or the fraction of the data that comes from a certain class, as applied to white-box Fully Connected Neural Networks (FCNNs). Because of their complexity and inscrutability, FCNNs have a particularly high risk of leaking unexpected information about their training sets; at the same time, this complexity makes extracting this information challenging. We develop techniques that reduce this complexity by noting that FCNNs are invariant under permutation of nodes in each layer. We develop our techniques using representations that capture this invariance and simplify the information extraction task. We evaluate our techniques on several synthetic and standard benchmark datasets and show that they are very effective at inferring various data properties. We also perform two case studies to demonstrate the impact of our attack. In the first case study we show that a classifier that recognizes smiling faces also leaks information about the relative attractiveness of the individuals in its training set. In the second case study we show that a classifier that recognizes Bitcoin mining from performance counters also leaks information about whether the classifier was trained on logs from machines that were patched for the Meltdown and Spectre attacks.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Quantifying Memorization Across Neural Language Models (2022, background): “, 2018) allow an adversary to detect the presence of a given example in a model’s training dataset, and other forms of data leakage permit an adversary to learn dataset properties (Ganju et al., 2018; Fredrikson et al., 2015).”
Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, background): “Specifically, an attacker could compromise the confidentiality of users’ private training or testing data via various attacks such as model inversion attacks [21, 22], membership inference attacks [40, 49, 53], and property inference attacks [1, 24].”
Privacy in Deep Learning: A Survey (2020, background): “Inference Parameter Inference Property Inference Access to Model Access to Model Output Membership Inference [13] # # # # Blackbox Logits Measuring Membership Privacy[35] # # # # Blackbox Logits ML-Leaks[36] # # # # Blackbox Logits The Natural Auditor [37] # # # # Blackbox Label LOGAN [38] # # # # B”
Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, methodology): “Attacks targeting data privacy include: the adversary inferring whether input examples were used to train the target model with membership inference attacks [38, 42, 59], learning global properties of training data with property inference attacks [12], reconstructing training data with model inversi”
Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks (2018, background): “Ganju et al. [26] designed a new property inference attack on fully connected networks.”
Attacking Graph-based Classification via Manipulating the Graph Structure (2019, background): “…examples [7,12,27,30,37,40,43,46,51,53,56, 68], poisoning attacks [8,18,31,38,67,70], privacy attacks for users (e.g., model inversion attacks [19,20], membership inference attacks [49,52], property inference attacks [3,21]), as well as model parameter and hyperparameter stealing attacks [60,61].”
Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning (2018, background): “class. This attack works on datasets where all of the members in the same classes are similar. No metric for leakage is used to measure privacy, except illustrating the generated images. Ganju et al. [26] designed a new property inference attack on fully connected networks. In this attack, the attac”
Gradient-Leaks: Understanding and Controlling Deanonymization in Federated Learning (2018, background): “…setting include: model stealing [52,62,77], membership inference [66,67,71] which identiﬁes if a particular example was used during training, attribute inference [28,56] to identify properties that holds true for subsets of data, and model inversion [27,41] to reconstruct training class exemplars.”
Model inversion attacks against collaborative inference (2019, background): “, property inference attacks [3, 11], membership inference attacks [18, 29, 30, 39, 41, 49], model inversion attacks [9, 10].”
When Machine Unlearning Jeopardizes Privacy (2020, background): “[20] proposed a property inference attack aiming at inferring general properties of the training data (such as the proportion of each class in the training data).”
Detecting AI Trojans Using Meta Neural Analysis (2019, methodology): “With white-box access to the target model, the features can be derived directly from the parameters of the model [19].”
Amnesiac Machine Learning (2020, background): “Property inference attacks are a subset of membership inference attacks that determine a general property of the training data, such as the ratio of training examples in each class (Ganju et al. 2018).”
FLAME: Taming Backdoors in Federated Learning (2022, background): “A number of attacks on FL have been proposed that aim to infer from parameters of a model the presence of a specific training sample in the training dataset (membership inference attacks) [41, 46, 52], properties of training samples (property inference attacks) [23,41], try to assess the proportion ”
A Survey of Privacy Attacks in Machine Learning (2020, background): “A common design pattern for a lot of supervised learning attacks is the use of shadow models and meta-models or attack-models [3, 18, 29, 40, 46, 51, 68, 97, 106, 108, 110, 116, 125].”
Towards Probabilistic Verification of Machine Unlearning (2020, background): “However, recent works have shown that ML models memorize sensitive information of training data [1–4], indicating serious privacy risks to individual user data.”
BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements (2020, background): “For instance, multiple works have explored the security and privacy threats of data used to train DNNmodels, such as membership inference attack [30, 35, 36], dataset reconstruction attack [33], and property inference attack [8, 12].”
Bag of Tricks for Training Data Extraction from Language Models (2023, background): “The objective of attribute inference attacks is to extract specific personal attributes such as locations, occupations, and interests from the model (Fredrikson et al., 2015; Gong & Liu, 2016; Ganju et al., 2018; Parisot et al., 2021).”
When Relaxations Go Bad: "Differentially-Private" Machine Learning (2019, methodology): “These attacks have been performed on HMM and SVM models [2] and neural networks [23].”
Eavesdrop the Composition Proportion of Training Labels in Federated Learning (2019, methodology): “Ganju et al. [47] construct an inference attack towards fully-connected neural networks, and they realize it by applying post-training techniques to a white-box model.”