Privacy Risk in Machine Learning: Analyzing the Connection to Overfitting
By Samuel Yeom, Irene Giacomelli, Matt Fredrikson, Somesh Jha. 2017
1,558 citations recorded by the source corpus.
Machine learning algorithms, when applied to sensitive data, pose a distinct threat to privacy. A growing body of prior work demonstrates that models produced by these algorithms may leak specific private information in the training data to an attacker, either through the models' structure or their observable behavior. However, the underlying cause of this privacy risk is not well understood beyond a handful of anecdotal accounts that suggest overfitting and influence might play a role. This paper examines the effect that overfitting and influence have on the ability of an attacker to learn information about the training data from machine learning models, either through training set membership inference or attribute inference attacks. Using both formal and empirical analyses, we illustrate a clear relationship between these factors and the privacy risk that arises in several popular machine learning algorithms. We find that overfitting is sufficient to allow an attacker to perform membership inference and, when the target attribute meets certain conditions about its influence, attribute inference attacks. Interestingly, our formal analysis also shows that overfitting is not necessary for these attacks and begins to shed light on what other factors may be in play. Finally, we explore the connection between membership inference and attribute inference, showing that there are deep connections between the two that lead to effective new attacks.
Subfields: Artificial Intelligence, Safety Research, Safety, Robustness & Interpretability, Privacy & Security
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Mohit Bansal
- Yoshua Bengio
- Quoc V. Le
- Luke S. Zettlemoyer
- Nicholas Carlini
- Jianfeng Gao
- Florian Tramèr
In Safety Research
- Prateek Mittal
- Liwei Song
- Reza Shokri
- Philip S. Yu
- Lichao Sun
- Gillian Dobbie
- Hongsheng Hu
- Zoran A. Salcic
In Safety, Robustness & Interpretability
- Nicholas Carlini
- Florian Tramèr
- Nicolas Papernot
- Reza Shokri
- Dawn Song
- Prateek Mittal
- Yang Zhang
- Liwei Song
In Privacy & Security
- Peter Kairouz
- Abhradeep Thakurta
- Vitaly Feldman
- Kunal Talwar
- Borja Balle
- Om Thakkar
- Rasmus Pagh
- H. Brendan McMahan
Citations in the corpus
- Quantifying Memorization Across Neural Language Models (2022, background): “For example, membership inference attacks (Shokri et al., 2017; Yeom et al., 2018) allow an adversary to detect the presence of a given example in a model’s training dataset, and other forms of data leakage permit an adversary to learn dataset properties (Ganju et al.”
- Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, background): “Past works on membership inference find that while overfitting is sufficient for performing membership inference, well-generalized models can also leak membership information (Yeom et al., 2018; Long et al., 2018).”
- Extracting Training Data from Large Language Models (2020, methodology): “Indeed, overfitting is a sufficient condition for privacy leakage [72] and many attacks work by exploiting overfitting [65].”
- Advances and Open Problems in Federated Learning (2019, background): “This can be approached theoretically as in [425], although this often yields loose, vacuous bounds for realistic models.”
- Auditing Differentially Private Machine Learning: How Private is Private SGD? (2020, methodology): “We evaluate both membership inference (MI, as used by [YGFJ18] and [JE19] and described in Appendix D) and our algorithms on three datasets: Fashion-MNIST (FMNIST), CIFAR10, and Purchase-100 (P100).”
- Privacy in Deep Learning: A Survey (2020, methodology): “Yeom et al. [15] also propose an attribute inference attack, built upon the same principle used for their membership inference attack, mentioned in Section 2.2.1.”
- Differential Privacy Has Disparate Impact on Model Accuracy (2019, background): “[38] show that poorly generalized models are more prone to leak training data.”
- Privacy Risks of Securing Machine Learning Models against Adversarial Examples (2019, methodology): “out the training data or the target model. Attacks targeting data privacy include: the adversary inferring whether input examples were used to train the target model with membership inference attacks [38, 42, 59], learning global properties of training data with property inference attacks [12], reco”
- Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks (2018, background): “[7] demonstrated the relationship between overfitting and membership inference attacks.”
- Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfer (2019, methodology): “Co-distillation approach is a method to use distillation on private training data with other parties [7], [28], however, without defending against data poisoning and inference attacks [48], [56], [22].”
- Auditing Data Provenance in Text-Generation Models (2018, background): “[35] formalize membership inference and theoretically show that overfitting is sufficient but not necessary.”
- Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning (2018, background): “[7] demonstrated the relationship between overfitting and membership inference attacks.”
- LOGAN: Membership Inference Attacks Against Generative Models (2017, background): “[70] investigate connections between membership inference and model inversion attacks against machine learning classifiers.”
- Encode, Shuffle, Analyze Privacy Revisited: Formalizations and Empirical Evaluation (2020, methodology): “Estimating lower bounds through membership inference attacks: We use the membership inference attack to measure the privacy leakage of a model [45], [46], [47], [48].”
- Machine Learning with Membership Privacy using Adversarial Regularization (2018, background): “The relation between some different definitions of membership privacy and differential privacy is analyzed in the literature [32, 51].”
- Measuring Forgetting of Memorized Training Examples (2022, background): “In membership inference [33, 21, 81, 69], an adversary infers whether or not a target example was contained in a model’s training set.”
- Information Leakage in Embedding Models (2020, background): “This subsequently led to a large body of work exploring this space [45, 48, 60, 61, 77].”
- Certified Data Removal from Machine Learning Models (2019, background): “This guarantees that membership-inference attacks (Yeom et al., 2018; Carlini et al., 2019) are unsuccessful on data that was removed from the model.”