The Unintended Consequences of Overfitting: Training Data Inference Attacks

By Samuel Yeom, Matt Fredrikson, Somesh Jha. 2017

41 citations recorded by the source corpus.

Machine learning algorithms that are applied to sensitive data pose a distinct threat to privacy. A growing body of prior work demonstrates that models produced by these algorithms may leak specific private information in the training data to an attacker, either through their structure or their observable behavior. However, the underlying cause of this privacy risk is not well understood beyond a handful of anecdotal accounts that suggest overfitting and influence might play a role. This paper examines the effect that overfitting and influence have on the ability of an attacker to learn information about training data from machine learning models, either through training set membership inference or model inversion attacks. Using both formal and empirical analyses, we illustrate a clear relationship between these factors and the privacy risk that arises in several popular machine learning algorithms. We find that overfitting is sufficient to allow an attacker to perform membership inference, and when certain conditions on the influence of certain features are present, model inversion attacks. Interestingly, our formal analysis also shows that overfitting is not necessary for these attacks, and begins to shed light on what other factors may be in play. Finally, we explore the connection between two types of attack, membership inference and model inversion, and show that there are deep connections between the two that lead to effective new attacks.

Subfields: Artificial Intelligence, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

Citations in the corpus

Property Inference Attacks on Fully Connected Neural Networks using Permutation Invariant Representations (2018, background): “Model inversion attacks [15, 16, 49] output some of the possible training data samples that an ML model could have been trained on.”
Understanding Membership Inferences on Well-Generalized Learning Models (2018, background): “In these attacks, the adversary controls the training algorithm and stealthily embeds information in the model [33, 39].”
Towards Measuring Membership Privacy (2017, background): “All these attacks can be summarized under the framework in [43].”
ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models (2018, background): “Following [36], many recent works have studied membership inference against machine learning from different angles [26], [27], [45].”
How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy (2023, background): “Additionally, new line of work demonstrated that it is possible to derive the bounds of success of a particular empirical class of attacks like membership inference from DP bounds (Yeom et al., 2017; Erlingsson et al., 2019b; Sablayrolles et al., 2019; Jayaraman et al., 2020), without having to do e”
Neural Network Inversion in Adversarial Setting via Background Knowledge Alignment (2019, background): “[75] show that prior model inversion attack [21, 22] and membership inference attack [65] are deeply related and are both sensitive to overfitting of the target model.”
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System? (2019, background): “Yeom et al. (2017) analyze how overﬁtting by Alice’s increases the risk privacy leakage; Long et al. (2018) showed that even a well-generalized model holds such risks in classiﬁcation problems, implying that overﬁtting by Alice is a sufﬁcient but not necessary condition for privacy leakage.”
Towards Demystifying Membership Inference Attacks (2018, background): “[45] investigates this belief and concludes that overfitting is not necessary for a model to show vulnerability to membership inference.”
Differentially Private Decoding in Large Language Models (2022, background): “The definition of DP is broader relative to the notion of memorization and therefore DP algorithms are usually robust to other threats such as data linkage attacks, side channel attacks, and membership inference attacks (Jagannatha et al., 2021; Yeom et al., 2017).”