Red-Teaming the Stable Diffusion Safety Filter

By Javier Rando, Daniel Paleka, David Lindner, Lennard Heim, Florian Tramèr. 2022

312 citations recorded by the source corpus.

Stable Diffusion is a recent open-source image generation model comparable to proprietary models such as DALLE, Imagen, or Parti. Stable Diffusion comes with a safety filter that aims to prevent generating explicit images. Unfortunately, the filter is obfuscated and poorly documented. This makes it hard for users to prevent misuse in their applications, and to understand the filter's limitations and improve it. We first show that it is easy to generate disturbing content that bypasses the safety filter. We then reverse-engineer the filter and find that while it aims to prevent sexual content, it ignores violence, gore, and other similarly disturbing content. Based on our analysis, we argue safety measures in future model releases should strive to be fully open and properly documented to stimulate security contributions from the community.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Safety, Robustness & Interpretability

Citations in the corpus

The Gradient of Generative AI Release: Methods and Considerations (2023, background): “Stable Diffusion’s safety filter was found to primarily prevent generations with sexually explicit content but not violence and gore [78], which is a normative judgment about the safety of both categories in generated images.”
On Evaluating Adversarial Robustness of Large Vision-Language Models (2023, background): “More examples such as attacking real-world image-text pairs are provided in our Appendix. accessible to the public typically include a safety checker to filter sensitive concepts and an invisible watermarking module to help identify fake content [69, 72, 108].”
Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models (2023, unknown): “If we set the similarity threshold to 0.21 inspired by Rando et al. (2022), then our adversarial samples might be detectable by some of the categories.”
MACE: Mass Concept Erasure in Diffusion Models (2024, methodology): “Existing research on preventing unwanted outputs from T2I models can be broadly grouped into four categories: post-image ﬁltering [41, 50], inference guidance [3, 59], retraining with the curated dataset [40, 52], and model ﬁnetuning [16, 17, 19, 25, 30, 39, 71].”
Unified Concept Editing in Diffusion Models (2023, background): “…training data or at inference time [12,27]; for example the Stable Diffusion implementation includes a “not safe for work” safety checker that returns a black image when an unsafe image is detected [3,22,32], and other work has addressed the issue in through image editing at inference time [35].”
Erasing Concepts from Diffusion Models (2023, methodology): “To limit the generation of unsafe images, the ﬁrst version was bundled with a simple NSFW ﬁlter to censor images if the ﬁlter is triggered [29], yet since both the code and model weights are publicly available, it is easy to disable the ﬁlter [43].”
Poisoning Web-Scale Training Datasets is Practical (2023, unknown): “We consider two poisoning attack objectives: (1) cause a particular image to be misclassified as some target label from ImageNet, and (2) cause a particular image to be classified as NSFW by the Stable Diffusion Safety Filter [78].”