Understanding Membership Inferences on Well-Generalized Learning Models

By Yunhui Long, Vincent Bindschaedler, Lei Wang, Diyue Bu, Xiaofeng Wang, Haixu Tang, Carl A. Gunter, Kai Chen. 2018

249 citations recorded by the source corpus.

Membership Inference Attack (MIA) determines the presence of a record in a machine learning model's training data by querying the model. Prior work has shown that the attack is feasible when the model is overfitted to its training data or when the adversary controls the training algorithm. However, when the model is not overfitted and the adversary does not control the training algorithm, the threat is not well understood. In this paper, we report a study that discovers overfitting to be a sufficient but not a necessary condition for an MIA to succeed. More specifically, we demonstrate that even a well-generalized model contains vulnerable instances subject to a new generalized MIA (GMIA). In GMIA, we use novel techniques for selecting vulnerable instances and detecting their subtle influences ignored by overfitting metrics. Specifically, we successfully identify individual records with high precision in real-world datasets by querying black-box machine learning models. Further we show that a vulnerable record can even be indirectly attacked by querying other related records and existing generalization techniques are found to be less effective in protecting the vulnerable instances. Our findings sharpen the understanding of the fundamental cause of the problem: the unique influences the training instance may have on the model.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, background): “Past works on membership inference ﬁnd that while overﬁtting is sufﬁcient for performing membership inference, well-generalized models can also leak membership information (Yeom et al., 2018; Long et al., 2018).”
Extracting Training Data from Large Language Models (2020, background): “Understanding why this happens is an important problem for future work [6,40].”
Exploiting Unintended Feature Leakage in Collaborative Learning (2018, background): “Membership inference attacks against aggregate statistics are well-known [14, 27, 50], and recent work demonstrated black-box membership inference against ML models [24, 34, 53, 58], as discussed in Section 10.”
Comprehensive Privacy Analysis of Deep Learning: Stand-alone and Federated Learning under Passive and Active White-box Inference Attacks (2018, background): “Multiple research papers address the problem of membership inference attack against model in a black-box setting [6], [7], [18].”
Contamination Attacks and Mitigation in Multi-Party Machine Learning (2018, background): “In multi-party machine learning, given a training record, predicting which party it belongs to is a form of a membership inference attack and has real privacy concerns (see [17, 25, 40]).”
Auditing Data Provenance in Text-Generation Models (2018, background): “nference to evaluate the tradeoff between test accuracy and membership privacy in differentially private ML models. Hayes et al. [11] study membership inference against generative models. Long et al. [19] show that well-generalized models can leak membership information, but the adversary must first”
Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learning (2018, background): “Multiple research papers have studied membership inference attacks in a black-box setting [6], [16], [7].”
Membership Inference Attacks on Machine Learning: A Survey (2021, background): “[181], Long et al.[123][124], Nasr et al.”
On the Importance of Difficulty Calibration in Membership Inference Attacks (2021, methodology): “This approach has been applied to the loss score for high-precision attack against well-generalized models (Long et al., 2018).”
The Natural Auditor: How To Tell If Someone Used Your Words To Train Their Model (2018, methodology): “Long et al. [22] showed that well-generalized models can leak membership information, but their attack requires the adversary to ﬁrst identify a handful of vulnerable records in the training dataset.”
Membership Leakage in Label-Only Exposures (2020, background): “Various recent research [23, 25, 31, 35, 36, 45, 46, 48, 49, 54, 57, 60] has shown that ML models are vulnerable to privacy attacks.”
When Machine Learning Meets Privacy (2020, methodology): “ing and non-training inputs. They also found that overfitting, the structure and type of the model are the main factors that cause a model to be vulnerable to membership inference attack. Long et al. [89] and Yeom et al. [173] investigated “the relationship between overfitting and privacy leakage”. ”
Membership Inference Attacks against Language Models via Neighbourhood Comparison (2023, background): “To account for this, previous work has introduced the idea of difficulty calibration mechanisms (Long et al., 2018; Watson et al., 2022), which aim to quantify the intrinsic complexity of a data sample (i.e., how much of an outlier the given sample is under the probability distribution of the…”
Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System? (2019, background): “s to prevent leaking information about data in the training set. The common enemy of both goals is overﬁtting. Yeom et al. (2017) analyze how overﬁtting by Alice’s increases the risk privacy leakage; Long et al. (2018) showed that even a well-generalized model holds such risks in classiﬁcation probl”
GAN-Leaks: A Taxonomy of Membership Inference Attacks against Generative Models (2019, background): “While numerous literature is dedicated to MIA against discriminative models [33, 45, 48, 50, 58, 60, 68], the attack on generative models has not received equal attention, despite its practical importance.”
White-box vs Black-box: Bayes Optimal Strategies for Membership Inference (2019, unknown): “A few recent works (Hayes et al., 2017; Shokri et al., 2017; Long et al., 2018)”
Systematic Evaluation of Privacy Risks of Machine Learning Models (2020, background): “For a target machine learning model, membership inference attacks aim to determine whether a given data point was used to train the model or not [23, 36, 39, 47].”