Membership Inference Attacks against Language Models via Neighbourhood Comparison

By Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schölkopf, Mrinmaya Sachan, Taylor Berg-Kirkpatrick. Annual Meeting of the Association for Computational Linguistics, 2023

339 citations recorded by the source corpus.

Membership Inference attacks (MIAs) aim to predict whether a data sample was present in the training data of a machine learning model or not, and are widely used for assessing the privacy risks of language models. Most existing attacks rely on the observation that models tend to assign higher probabilities to their training samples than non-training points. However, simple thresholding of the model score in isolation tends to lead to high false-positive rates as it does not account for the intrinsic complexity of a sample. Recent work has demonstrated that reference-based attacks which compare model scores to those obtained from a reference model trained on similar data can substantially improve the performance of MIAs. However, in order to train reference models, attacks of this kind make the strong and arguably unrealistic assumption that an adversary has access to samples closely resembling the original training data. Therefore, we investigate their performance in more realistic scenarios and find that they are highly fragile in relation to the data distribution used to train reference models. To investigate whether this fragility provides a layer of safety, we propose and evaluate neighbourhood attacks, which compare model scores for a given sample to scores of synthetically generated neighbour texts and therefore eliminate the need for access to the training data distribution. We show that, in addition to being competitive with reference-based attacks that have perfect knowledge about the training data distribution, our attack clearly outperforms existing reference-free attacks as well as reference-based attacks with imperfect knowledge, which demonstrates the need for a reevaluation of the threat model of adversarial attacks.

View Paper: https://www.semanticscholar.org/paper/cb754310302086dfbbcd098263200e2a03f65874

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Detecting Pretraining Data from Large Language Models (2023, methodology): “We follow the standard definition of the membership inference attack (MIA) by Shokri et al. (2016); Mattern et al. (2023).”
Security and Privacy Challenges of Large Language Models: A Survey (2024, unknown): “Differential privacy (DP) based techniques are also widely used to prevent privacy leakage by MIA [122], [178].”
Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents (2024, unknown): “Delimiters (Learn Prompting, 2023a; Mattern et al., 2023; Willison, 2022).”
LaMP: When Large Language Models Meet Personalization (2023, background): “Several studies have shown successful membership attacks against deep learning models (including LLMs) (Mattern et al., 2023; Shokri et al., 2017), thus us-ing personal and private data in fine-tuning may put the privacy of some users at risk.”
Proving Test Set Contamination in Black Box Language Models (2023, background): “2019; Mattern et al., 2023) as well as provide some evidence for test set contamination (Sainz et al., 2023; Golchin & Surdeanu, 2023).”
Formalizing and Benchmarking Prompt Injection Attacks and Defenses (2023, background): “Other attacks to LLM: Other attacks to LLMs (or LLM-Integrated Applications) include, but not limited to, privacy attacks [16, 30], poisoning attacks [13, 42, 47, 54], and adversarial prompts [56].”