Auditing Data Provenance in Text-Generation Models

By Congzheng Song, Vitaly Shmatikov. 2018

293 citations recorded by the source corpus.

To help enforce data-protection regulations such as GDPR and detect unauthorized uses of personal data, we develop a new model auditing technique that helps users check if their data was used to train a machine learning model. We focus on auditing deep-learning models that generate natural-language text, including word prediction and dialog generation. These models are at the core of popular online services and are often trained on personal data such as users' messages, searches, chats, and comments. We design and evaluate a black-box auditing method that can detect, with very few queries to a model, if a particular user's texts were used to train it (among thousands of other users). We empirically show that our method can successfully audit well-generalized models that are not overfitted to the training data. We also analyze how text-generation models memorize word sequences and explain why this memorization makes them amenable to auditing.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, background): “Membership inference can also be extended to audit models subject to data-protection laws (Song & Shmatikov, 2019).”
You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion (2020, methodology): “Further, attackers may use model auditing [57] to test if their repo is included.”
Red Teaming Language Models with Language Models (2022, methodology): “Data leakage can be used to infer the data used in training (“membership inference”; Shokri et al., 2017; Song and Shmatikov, 2019; Nasr et al., 2019; Hisamoto et al., 2020; Carlini et al., 2021), helping adversaries clone private, commercial LMs, violating intellectual property (Ateniese et al.”
Practical and Private (Deep) Learning without Sampling or Shuffling (2021, background): “[66] Congzheng Song and Vitaly Shmatikov.”
The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation (2021, background): “…individual client updates, the server is still able to learn the sum of the updates, which itself may leak potentially sensitive information (Melis et al., 2019; Carlini et al., 2019; Song & Shmatikov, 2019a; Dwork et al., 2015; Song & Shmatikov, 2019b; Nasr et al., 2021; Shokri et al., 2017).”
Tempered Sigmoid Activations for Deep Learning with Differential Privacy (2020, background): “…learning models easily memorize sensitive, personal, or private data that was used in their training, and models may in practice disclose this data—as demonstrated by membership inference attacks (Shokri et al. 2017) and secret extraction results (Song and Shmatikov 2019; Carlini et al. 2019).”
Privacy in Deep Learning: A Survey (2020, background): “box Logits Measuring Membership Privacy[40] # # # # Blackbox Logits ML-Leaks[41] # # # # Blackbox Logits The Natural Auditor [42] # # # # Blackbox Label LOGAN [43] # # # # Both Logits Data Provenance [44] # # # # Blackbox Logits Privacy Risk in ML [17] # # # Whitebox Logits+Auxilary Fredrikson et al”
Generative Models for Effective ML on Private, Decentralized Datasets (2019, background): “Note that ( , δ) DP upper bounds are often loose, and so can be complemented by direct measurement of memorization, as in Carlini et al. (2018); Song & Shmatikov (2019).”
Theoretical Guarantees for Model Auditing with Finite Adversaries (2019, background): “Thus, DP implementations can be complemented by direct measurement of memorization, as in [2, 7, 13].”
Impact of Pretraining Term Frequencies on Few-Shot Reasoning (2022, background): “Although frequent terms are more likely to be memorized, data privacy researchers have also shown that LMs may memorize sensitive sequences occurring in training data (e.g., social security and credit card numbers), even if they are rare (Carlini et al., 2019; Song & Shmatikov, 2019).”
Information Leakage in Embedding Models (2020, methodology): “Membership inference attacks (MIA) have first been studied against black-box supervised classification models [66], and later on generative models and language models [23, 67].”
A Survey on Differential Privacy for Unstructured Data Content (2022, methodology): “As for privacy evaluations, this work employs query scrambling methods [89] to evaluate the realistic privacy loss.”
Large Language Model Alignment: A Survey (2023, result): “Attribute Inference Attacks infers data ownership and privacy attributes by comparing the performance of a target model with that of similar models (Song and Shmatikov, 2019; Hisamoto et al., 2020; Mireshghallah et al., 2022).”