Extracting Training Data from Diffusion Models

By Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, Eric Wallace. 2023

986 citations recorded by the source corpus.

Image diffusion models such as DALL-E 2, Imagen, and Stable Diffusion have attracted significant attention due to their ability to generate high-quality synthetic images. In this work, we show that diffusion models memorize individual images from their training data and emit them at generation time. With a generate-and-filter pipeline, we extract over a thousand training examples from state-of-the-art models, ranging from photographs of individual people to trademarked company logos. We also train hundreds of diffusion models in various settings to analyze how different modeling and data decisions affect privacy. Overall, our results show that diffusion models are much less private than prior generative models such as GANs, and that mitigating these vulnerabilities may require new advances in privacy-preserving training.

Subfields: Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Memorization of training examples remains a critical security and privacy concern [49, 50].”
Foundation Models and Fair Use (2023, background): “And others have found that it is possible in some circumstances to extract training data from image generation foundation models (Somepalli et al., 2022; Carlini et al., 2023).”
Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models (2023, background): “[7, 60] have successfully retrieved samples from Stable Diffusion that are highly faithful to real training examples.”
On Provable Copyright Protection for Generative Models (2023, background): “” Gillotte [2020] studies copyright infringement in AI-generated artworks and concludes that regarding the training phase “an engineer may use copyrighted works to train an Al program to generate artwork without incurring infringement liability.”
Ablating Concepts in Text-to-Image Diffusion Models (2023, methodology): “To measure the effectiveness of our method in ablating memorized images, following previous works [50, 10], we use SSCD [50] model to measure the percentage of generated images having similarity with the memorized image greater than a threshold.”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, methodology): “[247] further explored the privacy vulnerabilities of state-of-the-art diffusion models by leveraging a generate-and-filter pipeline to extract over a thousand training examples from the models.”
A Categorical Archive of ChatGPT Failures (2023, neutral): “This is inherently tied to the question of plagiarism and copyright ( e.g. [8]).”
Diffusion models in medical imaging: A comprehensive survey (2022, background): “[191] conducted extensive experiments to evaluate the privacy concerns associated with generative diffusion models.”
Art and the science of generative AI (2023, background): “How often do these models directly copy elements from the training data, versus creating entirely new works [71, 65]?”
CAMEL: Communicative Agents for "Mind" Exploration of Large Scale Language Model Society (2023, methodology): “Recent works have proposed innovative methods for extracting training data from both large language models [11] diffusion models [12].”
DIRE for Diffusion-Generated Image Detection (2023, background): “One typical low-level analysis of images is noise pattern analysis 3 , which is usually regular and corresponds to the shape of objects in real scenarios.”
Regulating ChatGPT and other Large Generative AI Models (2023, background): “Third, we highlight key data protection risks under the GDPR, with a particular focus on model inversion [27-29].”
AI Art and its Impact on Artists (2023, background): “In fact, while image generators have been shown to even memorize their data and can output almost exact replicas of images from their training set under certain conditions [27, 117], as artist Karla Ortiz writes, artists’ styles are so unique to them,”
MACE: Mass Concept Erasure in Diffusion Models (2024, methodology): “An alternative method for safeguarding images against imitation or memorization [8, 65] by T2I models involves an additional step of applying adversarial perturbations to photographs or artworks before they are posted online.”
Stable Bias: Evaluating Societal Representations in Diffusion Models (2023, neutral): “…understand their inner workings, such as the functioning of their safety filters [51], the structure of their semantic space [14], the extent to which they generate stereotypical representations of given demographics [10, 19, 44] and memorize and regenerate specific training examples [16, 62].”