The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

By Nicholas Carlini, Chang Liu, Ú. Erlingsson, Jernej Kos, Dawn Song. 2018

1,449 citations recorded by the source corpus.

This paper describes a testing methodology for quantitatively assessing the risk that rare or unique training-data sequences are unintentionally memorized by generative sequence models---a common type of machine-learning model. Because such models are sometimes trained on sensitive data (e.g., the text of users' private messages), this methodology can benefit privacy by allowing deep-learning practitioners to select means of training that minimize such memorization. In experiments, we show that unintended memorization is a persistent, hard-to-avoid issue that can have serious consequences. Specifically, for models trained without consideration of memorization, we describe new, efficient procedures that can extract unique, secret sequences, such as credit card numbers. We show that our testing strategy is a practical and easy-to-use first line of defense, e.g., by describing its application to quantitatively limit data exposure in Google's Smart Compose, a commercial text-completion neural network trained on millions of users' email messages.

Subfields: Artificial Intelligence, Information Systems, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Safety, Robustness & Interpretability

Citations in the corpus

Ethical and social risks of harm from Language Models (2021, background): “This ’unintended memorization’ of training data can occur even when there is not overfitting in the traditional statistical sense (Carlini et al., 2019), and can be observed serendipitously when sampling from LMs even without any form of "malicious" prompting (Carlini et al., 2021).”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “, 2020)? The non-trivial amounts of toxicity generated by DAPT suggest that perhaps language models may be “memorizing” the toxicity in pretraining data (Carlini et al., 2019) or that toxic examples may be more salient for the model and hence harder to unlearn (Koh and Liang, 2017).”
Quantifying Memorization Across Neural Language Models (2022, methodology): “, 2020; Ziegler, 2021), or in other work, synthetically injected “canaries” (Carlini et al., 2019; Henderson et al., 2017; Thakkar et al., 2020; Thomas et al., 2020).”
Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, methodology): “In this work we focus on generation-based memorization, where a generative model leaks information by generating exact samples from its training data (Carlini et al., 2019).”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “Studies indicate that LMs might memorize or leak these details (Carlini et al., 2019, 2021), and under certain techniques, attackers can decode private data from model inferences (Li et al., 2022a; Pan et al., 2020; Song and Raghunathan, 2020).”
Pretraining Language Models with Human Preferences (2023, background): “PII LMs sometimes generate text that occurs verbatim in their training data (Carlini et al., 2019; Perez et al., 2022).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, methodology): “These biases can result in toxicity being associated with certain words, such as profanities or identity terms (Dinan et al., 2019b; Dixon et al., 2018), or language varieties, such as African American English (AAE) (Liu et al., 2019; Sap et al., 2019).”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “Similar privacy problems are observed for LLMs (e.g. Nasr et al., 2019; Shokri et al., 2017; Carlini et al., 2019, 2020).”
Extracting Training Data from Large Language Models (2020, methodology): “Prior work shows that (1000× smaller) language models potentially memorize training data in semi-realistic settings [6].”
Red Teaming Language Models with Language Models (2022, methodology): “LMs generate text from the training data, posing many risks (see Carlini et al., 2019, for an overview).”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “Building on the extensive literature on memorization in large language models (Carlini et al., 2019; 2021; Hu et al., 2022), we ask the following question: does the location of a particular sequence in the training dataset influence the likelihood of it being memorized?”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “At the same time, datasets curated from the Web could come with risks such as unsuitable content (adult content, profanity) and unintended privacy leakage [57, 17, 18].”
Foundation Models and Fair Use (2023, background): “First, machine learning with DP has often been reported to suffer from high computational costs (Carlini et al., 2019).”
Modifying Memories in Transformer Models (2020, background): “…modiﬁcation further requires the model to memorize new facts that conﬂict with previously learned facts, posing new challenges to existing continual learning approaches, e.g., we may need to update the Gradient Episodic Memory (Lopez-Paz and Ranzato, 2017) or the Concep-tors (Liu et al., 2019).”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, background): “While this kind of documentation is far from standard practice, it is becoming increasingly common given recent calls for better documentation (Rogers, 2021; Bender et al., 2021) as well as empirical studies on data memorization in language models (Carlini et al., 2019, 2022).”
The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregation (2021, background): “…individual client updates, the server is still able to learn the sum of the updates, which itself may leak potentially sensitive information (Melis et al., 2019; Carlini et al., 2019; Song & Shmatikov, 2019a; Dwork et al., 2015; Song & Shmatikov, 2019b; Nasr et al., 2021; Shokri et al., 2017).”
Differentially Private Learning Needs Better Features (or Much More Data) (2020, background): “…under the framework of differential privacy (DP) (Dwork et al., 2006b; Chaudhuri et al., 2011; Shokri & Shmatikov, 2015; Abadi et al., 2016) can protect deployed classifiers against unintentional leakage of private training data (Shokri et al., 2017; Song et al., 2017; Carlini et al., 2019; 2020).”