Analyzing Information Leakage of Updates to Natural Language Models

By Santiago Zanella-B'eguelin, Lukas Wutschitz, Shruti Tople, Victor Rühle, Andrew J. Paverd, Olga Ohrimenko, Boris Köpf, Marc Brockschmidt. 2019

147 citations recorded by the source corpus.

To continuously improve quality and reflect changes in data, machine learning applications have to regularly retrain and update their core models. We show that a differential analysis of language model snapshots before and after an update can reveal a surprising amount of detailed information about changes in the training data. We propose two new metrics---differential score and differential rank---for analyzing the leakage due to updates of natural language models. We perform leakage analysis using these metrics across models trained on several different datasets using different methods and configurations. We discuss the privacy implications of our findings, propose mitigation strategies and evaluate their effect.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Extracting Training Data from Large Language Models (2020, background): “Prior work shows that (100 × to 1000 × smaller) language models potentially memorize training data in semi-realistic settings [8,73].”
Privacy in Deep Learning: A Survey (2020, methodology): “inversion attack on online-learning, using a generative adversarial network and based on the difference between a model, before and after a gradient update. In the same direction, Brockschmidt et al. [57] demonstrate the information leakage of updates to language models. More recently, He et al. [47”
Counterfactual Memorization in Neural Language Models (2021, methodology): “When the training data is not available, heuristic-based methods comparing language model perplexities are used to predict whether a generation contains memorized content (Carlini et al., 2019; Thakkar et al., 2020; Thomas et al., 2020; Carlini et al., 2020; Zanella-Béguelin et al., 2020).”
What Does it Mean for a Language Model to Preserve Privacy? (2022, background): “Training data extraction refers to the risk of partially extracting training samples by interacting with a trained language model [12, 13, 80, 99].”
Measuring Forgetting of Memorized Training Examples (2022, background): “Both of these privacy attacks have been shown to perform better when models are updated repeatedly, and when these repeated updates are all released to the adversary [66, 82, 39].”
Security and Privacy Challenges of Large Language Models: A Survey (2024, unknown): “Limiting the accessibility to the model and generating limited prediction results might be another option [137].”
Machine Unlearning of Features and Labels (2021, background): “If this data contains private information like credit card numbers or telephone numbers, this may become a severe privacy issue [13, 53].”
Do Language Models Plagiarize? (2022, background): “There is a growing body of literature that aims to study the memorization of neural LMs by recovering texts in the training corpus [31, 47] or extracting artifcially injected canaries [37, 58].”
A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication (2021, background): “…also proposed that: 1) it is possible to reveal details from an initial dataset even when a model was subsequently retrained on a redacted version [104]; 2) adversarial querying of a trained language model can recover individual training examples [22] and 3) "radioactive" data tracing is possible…”
When Machine Unlearning Jeopardizes Privacy (2020, background): “This is realistic as the target black-box model can be queried at any time, such as in the setting of MLaaS, and all of the query results can be stored locally by the adversary; this also follows the assumption of previous works [7, 59, 60, 64, 68].”
You Don’t Know My Favorite Color: Preventing Dialogue Representations from Revealing Speakers’ Private Personas (2022, unknown): “Furthermore, given black-box access to a language model’s pre-train and finetune stages, Zanella-Béguelin et al. (2020) showed that sensitive sequences of the fine-tuning dataset can be extracted.”
Detecting Pretraining Data from Large Language Models (2023, background): “…privacy vulnerabilities within machine learning models and in verifying the accurate implementation of privacy-preserving mechanisms (Jayaraman & Evans, 2019; Jagielski et al., 2020; Zanella-Béguelin et al., 2020; Nasr et al., 2021; Huang et al., 2022; Nasr et al., 2023; Steinke et al., 2023).”
Analyzing Leakage of Personally Identifiable Information in Language Models (2023, background): “Research has shown the feasibility of extracting different types of information including individual sentences [9, 28], inserted canaries [8, 50, 68] as well as n -grams [42].”