Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inference

By Klas Leino, Matt Fredrikson. USENIX Security Symposium, 2019

327 citations recorded by the source corpus.

Membership inference (MI) attacks exploit the fact that machine learning algorithms sometimes leak information about their training data through the learned model. In this work, we study membership inference in the white-box setting in order to exploit the internals of a model, which have not been effectively utilized by previous work. Leveraging new insights about how overfitting occurs in deep neural networks, we show how a model's idiosyncratic use of features can provide evidence for membership to white-box attackers---even when the model's black-box behavior appears to generalize well---and demonstrate that this attack outperforms prior black-box methods. Taking the position that an effective attack should have the ability to provide confident positive inferences, we find that previous attacks do not often provide a meaningful basis for confidently inferring membership, whereas our attack can be effectively calibrated for high precision. Finally, we examine popular defenses against MI attacks, finding that (1) smaller generalization error is not sufficient to prevent attacks on real models, and (2) while small-$\epsilon$-differential privacy reduces the attack's effectiveness, this often comes at a significant cost to the model's accuracy; and for larger $\epsilon$ that are sometimes used in practice (e.g., $\epsilon=16$), the attack can achieve nearly the same accuracy as on the unprotected model.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/a7da2f44aa205b9b99d8c04d3dfa51c2840b6605

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

On the Importance of Difficulty Calibration in Membership Inference Attacks (2021, methodology): “The efﬁcacy of membership inference attacks has been improved signiﬁcantly since the ﬁrst attempts (Salem et al., 2018; Sablayrolles et al., 2019; Leino & Fredrikson, 2020).”
Label-Only Membership Inference Attacks (2020, background): “We note that studying membership inference attacks with white-box model access [39] has merits (e.”
Enhanced Membership Inference Attacks against Machine Learning Models (2021, background): “[30], a substantial body of literature Leino and Fredrikson [15], Long et al.”
Membership Inference Attacks on Machine Learning: A Survey (2021, background): “Thus, Leino and Fredrikson propose an effective white-box MIA that does not require any of the target model’s training members.”
Bag of Tricks for Training Data Extraction from Language Models (2023, methodology): “Several defense methods based on differential privacy (Leino & Fredrikson, 2020; Naseri et al., 2020; Choquette-Choo et al., 2021), data pruning (Wang et al., 2021), data augmentation (Kaya & Dumitras, 2021) and causal inference (Tople et al., 2020) have been proposed to mitigate this vulnerability.”
Detecting Pretraining Data from Large Language Models (2023, background): “This problem is an instance of Membership Inference Attacks (MIAs), which was initially proposed by Shokri et al. (2016).”
Membership Inference Attacks against Language Models via Neighbourhood Comparison (2023, background): “…assume to only have access to model confidence or loss scores (Yeom et al., 2018; Sablayrolles et al., 2019; Jayaraman et al., 2020; Watson et al., 2022), others exploit additional information such as model parameters (Leino and Fredrikson, 2020) or training loss trajectories (Liu et al., 2022).”
Dataset Inference: Ownership Resolution in Machine Learning (2021, background): “MI attacks are shown to work in white-(Leino & Fredrikson, 2020; Sablayrolles et al., 2019) as well as black-box scenarios against various target models including generative models (Hayes et al., 2019).”
Do Language Models Plagiarize? (2022, background): “memorization of neural language models by recovering texts in the training text corpus (Salem et al., 2020; Kharitonov et al., 2021; Leino and Fredrikson, 2020) or extracting artificially injected canaries (Henderson et al.”
A Survey of Privacy Attacks in Machine Learning (2020, background): “In a white-box setting also, Leino and Fredrikson [66] showed that the learned features of the target model capture differences between the training data distribution and the general population.”
Systematic Evaluation of Privacy Risks of Machine Learning Models (2020, methodology): “[22] suggest using the metric of prediction correctness as a sign of being a member or not.”
Membership Inference Attacks From First Principles (2021, methodology): “The simplest method to evaluate attack efficacy is through a standard “accuracy” metric that measures how often an attack correctly predicts membership on a balanced dataset of members and non-members [6, 18, 33, 46, 56, 60, 61, 66, 68, 70].”