Model Inversion Attacks that Exploit Confidence Information and Basic Countermeasures
By Matt Fredrikson, Somesh Jha, Thomas Ristenpart. 2015
3,376 citations recorded by the source corpus.
Machine-learning (ML) algorithms are increasingly utilized in privacy-sensitive applications such as predicting lifestyle choices, making medical diagnoses, and facial recognition. In a model inversion attack, recently introduced in a case study of linear classifiers in personalized medicine by Fredrikson et al., adversarial access to an ML model is abused to learn sensitive genomic information about individuals. Whether model inversion attacks apply to settings outside theirs, however, is unknown. We develop a new class of model inversion attack that exploits confidence values revealed along with predictions. Our new attacks are applicable in a variety of settings, and we explore two in depth: decision trees for lifestyle surveys as used on machine-learning-as-a-service systems and neural networks for facial recognition. In both cases confidence values are revealed to those with the ability to make prediction queries to models. We experimentally show attacks that are able to estimate whether a respondent in a lifestyle survey admitted to cheating on their significant other and, in the other context, show how to recover recognizable images of people's faces given only their name and access to the ML model. We also initiate experimental exploration of natural countermeasures, investigating a privacy-aware decision tree training algorithm that is a simple variant of CART learning, as well as revealing only rounded confidence values. The lesson that emerges is that one can avoid these kinds of MI attacks with negligible degradation to utility.
Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Yoshua Bengio
- Mohit Bansal
- Nicholas Carlini
- Quoc V. Le
- Bernt Schiele
- Nicolas Papernot
- Dacheng Tao
In Radiology, Nuclear Medicine and Imaging
- Reza Shokri
- Congzheng Song
- Nicolas Papernot
- Florian Tramèr
- Boi Faltings
- Aleksei Triastcyn
- Milad Nasr
- Bingsheng He
In Safety, Robustness & Interpretability
- Nicholas Carlini
- Nicolas Papernot
- Florian Tramèr
- Reza Shokri
- Dawn Song
- Prateek Mittal
- Bo Li
- Yang Zhang
Citations in the corpus
- Quantifying Memorization Across Neural Language Models (2022, background): “, 2018) allow an adversary to detect the presence of a given example in a model’s training dataset, and other forms of data leakage permit an adversary to learn dataset properties (Ganju et al., 2018; Fredrikson et al., 2015).”
- Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, background): “Early model inversion attacks use a trained model and non-sensitive features of a training sample to reconstruct that sample’s sensitive features (Fredrikson et al., 2015).”
- This Face Does Not Exist... But It Might Be Yours! Identity Leakage in Generative Models (2020, background): “This is further explained by Fredrikson et al.[8] who focus on adversarial attacks.”
- Towards the Science of Security and Privacy in Machine Learning (2016, methodology): “Potential risks are adversaries performing membership test (to know whether an individual is in a dataset or not) [36], recovering of partially known inputs (use the model to complete an input vector with the most likely missing bits), and extraction of the training data using the model’s prediction”
- Deep Learning with Differential Privacy (2016, background): “demonstrated a model-inversion attack that recovers images from a facial recognition system [26].”
- Membership Inference Attacks Against Machine Learning Models (2016, methodology): “Inferring information about the model’s training dataset should not be confused with techniques such as model inversion that use a model’s output on a hidden input to infer something about this input [17] or to extract features that characterize one of the model’s classes [16].”
- Advances and Open Problems in Federated Learning (2019, background): “For classic (non-federated) models of computation, understanding a model’s susceptibility to attacks is an active and challenging research area [167, 357, 91, 293].”
- Local Model Poisoning Attacks to Byzantine-Robust Federated Learning (2019, background): “Specifically, an attacker could compromise the confidentiality of users’ private training or testing data via various attacks such as model inversion attacks [21, 22], membership inference attacks [40, 49, 53], and property inference attacks [1, 24].”
- Federated Learning With Differential Privacy: Algorithms and Performance Analysis (2019, background): “For example, authors in [36] demonstrated a model-inversion attack that recovers images from a facial recognition system.”
- Deep Models Under the GAN: Information Leakage from Collaborative Deep Learning (2017, methodology): “This adversarial influence is what makes our attack more effective than, for instance, just applying model inversion attacks [26] against the final trained model.”
- Semi-supervised Knowledge Transfer for Deep Learning from Private Training Data (2016, methodology): “To protect the privacy of training data, this paper improves upon a specific, structured application of the techniques of knowledge aggregation and transfer (Breiman, 1994), previously explored by Nis-sim et al. (2007), Pathak et al. (2010), and particularly Hamm et al. (2016).”
- Wild Patterns: Ten Years After the Rise of Adversarial Machine Learning (2017, background): “…here, we refer the reader to few practical examples of such attacks reported to date, including model inversion attacks aimed to steal machine learning models and earlier hill-climbing attacks against biometric systems used to steal the face and fingerprint templates of their users [28, 62, 67–70].”
- DISCO: Dynamic and Invariant Sensitive Channel Obfuscation for deep neural networks (2020, background): “Conventionally, research in privacy-aware machine learning has primarily focused on protecting training data from membership inference [4] and model inversion attacks [5], when i) training data is distributed over clients and ii) computation of training the model is out-sourced.”
- FedML: A Research Library and Benchmark for Federated Machine Learning (2020, background): “While the FL framework facilitates data privacy [89] by keeping data locally available to the users and only requiring communication for model updates, users may still be concerned about partial leakage of their data which may be inferred from the communicated model (see, e.g. , [112]).”