A Methodology for Formalizing Model-Inversion Attacks

By Xi Wu, Matt Fredrikson, Somesh Jha, Jeffrey F. Naughton. 2016

191 citations recorded by the source corpus.

Confidentiality of training data induced by releasing machine-learning models, and has recently received increasing attention. Motivated by existing MI attacks and other previous attacks that turn out to be MI "in disguise," this paper initiates a formal study of MI attacks by presenting a game-based methodology. Our methodology uncovers a number of subtle issues, and devising a rigorous game-based definition, analogous to those in cryptography, is an interesting avenue for future work. We describe methodologies for two types of attacks. The first is for black-box attacks, which consider an adversary who infers sensitive values with only oracle access to a model. The second methodology targets the white-box scenario where an adversary has some additional knowledge about the structure of a model. For the restricted class of Boolean models and black-box attacks, we characterize model invertibility using the concept of influence from Boolean analysis in the noiseless case, and connect model invertibility with stable influence in the noisy case. Interestingly, we also discovered an intriguing phenomenon, which we call "invertibility interference," where a highly invertible model quickly becomes highly non-invertible by adding little noise. For the white-box case, we consider a common phenomenon in machine-learning models where the model is a sequential composition of several sub-models. We show, quantitatively, that even very restricted communication between layers could leak a significant amount of information. Perhaps more importantly, our study also unveils unexpected computational power of these restricted communication channels, which, to the best of our knowledge, were not previously known.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability, Privacy & Security

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

In Privacy & Security

Citations in the corpus

Privacy in Deep Learning: A Survey (2020, background): “Model inversion and attribute inference attacks are against attribute privacy, where an adversary tries to infer sensitive attributes of given data instances from a released model and the instance’s non-sensitive attributes [47].”
Differentially Private Learning with Adaptive Clipping (2019, neutral): “However, recent work has shown that it is possible to extract detailed information about individual training examples using only the parameters of a trained model (Fredrikson et al., 2015; Wu et al., 2016; Shokri et al., 2017; Carlini et al., 2018; Melis et al., 2018).”
DeepXplore: Automated Whitebox Testing of Deep Learning Systems (2017, background): “Recently, the security of machine learning has drawn significant attention from the researchers in both machine learning [18, 33, 50] and security [7, 14, 15, 36, 42, 51, 58] communities.”
Towards Practical Differentially Private Convex Optimization (2019, background): “However, as many recent works [1], [2], [3] indicate, a model can leak information about the sensitive data it was trained on, even though the data might have never been made public.”
Advances in adversarial attacks and defenses in computer vision: A survey (2021, methodology): “For example, [151] uses the notion of influence from Boolean analysis to characterized inversion of Boolean functions.”
A Survey of Privacy Attacks in Machine Learning (2020, background): “The papers are discussed in two sections: attacks on centralized learning and attacks on distributed learning.”
Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting (2017, methodology): “We formalize quantitative advantage measures that capture the privacy risk to training data posed by two types of attack, namely membership inference [6], [7] and attribute inference [3], [4], [5], [8].”
Differential Privacy for Deep and Federated Learning: A Survey (2022, background): “Since, a malicious user (or the server) could reconstruct the users’ local dataset using only the local gradient sent to the server [10], [122], [123].”
Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference (2019, background): “Many compelling applications of machine learning involve the collection and processing of sensitive personal data, giving rise to concerns about privacy [2, 4, 7, 10, 11, 26, 33, 38, 45, 46].”
Adversarial Neural Network Inversion via Auxiliary Knowledge Alignment (2019, methodology): “[45] propose a methodology to formalize such attribute inference attacks.”
The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networks (2019, background): “For instance, [22] characterized model invertibility for Boolean functions using the concept of influence from Boolean analysis; [26] formalized the risk that the model poses specifically to individuals in the training data and shows that the risk increases with the degree of overfitting of the mode”
A Survey of Machine and Deep Learning Methods for Internet of Things (IoT) Security (2018, methodology): “The last possible attack is inversion, which exploits the application program interfaces presented to the users by the current ML platform to collect roughly the necessary information about the pre-trained ML models [280], [285].”
Differentially Private Learning with Adaptive Clipping (2019, background): “Moreover, many recent works [10, 27, 24, 5, 20] have shown that it is possible to extract sensitive information about the training data just from the parameters of a trained model.”
Trojaning Attack on Neural Networks (2018, methodology): “Model inversion is another important line of works in adversarial machine learning [27, 28, 56, 58].”
When Relaxations Go Bad: "Differentially-Private" Machine Learning (2019, unknown): “1 With additional access to conﬁdence scores of the model (noted as white-box information by Wu et al. [71]), more complex tasks have been performed, such as recovering faces from the training data [20].”
The Unintended Consequences of Overfitting: Training Data Inference Attacks (2017, methodology): “This approach is distinct from the way that inversion advantage has largely been characterized in prior work [4, 3, 5], where the priority was on empirically measuring advantage relative to a “simulator” who is not given access to the model.”