Membership Inference Attacks From First Principles

By Nicholas Carlini, Steve Chien, Milad Nasr, Shuang Song, Andreas F. Terzis, Florian Tramèr. 2021

1,168 citations recorded by the source corpus.

A membership inference attack allows an adversary to query a trained machine learning model to predict whether or not a particular example was contained in the model’s training dataset. These attacks are currently evaluated using average-case “accuracy” metrics that fail to characterize whether the attack can confidently identify any members of the training set. We argue that attacks should instead be evaluated by computing their true-positive rate at low (e.g., ≤ 0.1%) false-positive rates, and find most prior attacks perform poorly when evaluated in this way. To address this we develop a Likelihood Ratio Attack (LiRA) that carefully combines multiple ideas from the literature. Our attack is $10\times$ more powerful at low false-positive rates, and also strictly dominates prior attacks on existing metrics.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, methodology): “We demonstrate that the membership inference methods from Carlini et al. (2021b) are correlated with the number of duplicates of a sequence in the training data.”
Counterfactual Memorization in Neural Language Models (2021, methodology): “To mitigate this issue, we explore an alternative measure by taking the logit on the per-token accuracy [Carlini et al., 2021].”
Measuring Forgetting of Memorized Training Examples (2022, methodology): “To achieve this, our work leverages recent progress in auditing of privacy in ML [9, 76].”
Membership Inference Attacks against Language Models via Neighbourhood Comparison (2023, methodology): “when evaluated for very low false positive rates (Carlini et al., 2021a; Watson et al., 2022).”
TOFU: A Task of Fictitious Unlearning for LLMs (2024, neutral): “With unlimited resources, one could test the ( ε , δ ) -unlearning condition of indistinguishability (Bourtoule et al., 2021; Sekhari et al., 2021) by training many models and performing hypothesis tests—and this is done in practice when feasible (Carlini et al., 2022; Pawelczyk et al., 2023).”
Analyzing Leakage of Personally Identifiable Information in Language Models (2023, background): “We did not consider high-precision/low-recall attacks [10] and further research is needed to explore their potential impact and effectiveness.”
Extracting Training Data from Diffusion Models (2023, methodology): “LiRA finishes the initialization process by fitting Gaussians NIN to the IN set and NOUT to OUT set of losses.”
SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation (2023, background): “Examples of such methods include differential privacy (DP)-enforced unlearning and certified data removal (Guo et al., 2019; Chien et al., 2022).”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “These studies have explored various membership inference attacks by analyzing the label [42], determining the threshold [115, 28, 95], developing a generalized formulation [269], among other methods.”
Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models (2022, background): “In the field of membership inference attacks, one seeks to determine whether a chosen image was part of the training set [8, 32, 63, 64].”
Detecting Pretraining Data from Large Language Models (2023, methodology): “MIA methods for finetuning (Carlini et al., 2022; Watson et al., 2022) usually calibrate the target model probabilities of an example using a shadow reference model that is trained on a similar data distribution.”
Unlocking High-Accuracy Differentially Private Image Classification through Scale (2022, background): “Alongside other results (Carlini et al., 2021a; Choquette-Choo et al., 2021; Liu et al., 2021), these studies demonstrate that models trained on sensitive datasets present a signiﬁcant privacy risk.”
Enhanced Membership Inference Attacks against Machine Learning Models (2021, methodology): “We also show that our systematic efforts result in attacks that are more powerful than [2] (higher AUC scores with comparable TPR at small FPR) in Section 5.”
Towards Unbounded Machine Unlearning (2023, methodology): “Computing the ‘confidence’ Consistent with [Carlini et al., 2022], the confidence of an example (x, y) (where y denotes the ground-truth class label) is defined as softmax(f(x))[y].”