RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models

By Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, Noah A. Smith. 2020

1,775 citations recorded by the source corpus.

Pretrained neural language models (LMs) are prone to generating racist, sexist, or otherwise toxic language which hinders their safe deployment. We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration. We create and release RealToxicityPrompts, a dataset of 100K naturally occurring, sentence-level prompts derived from a large corpus of English web text, paired with toxicity scores from a widely-used toxicity classifier. Using RealToxicityPrompts, we find that pretrained LMs can degenerate into toxic text even from seemingly innocuous prompts. We empirically assess several controllable generation methods, and find that while data- or compute-intensive methods (e.g., adaptive pretraining on non-toxic data) are more effective at steering away from toxicity than simpler solutions (e.g., banning “bad” words), no current method is failsafe against neural toxic degeneration. To pinpoint the potential cause of such persistent toxic degeneration, we analyze two web text corpora used to pretrain several LMs (including GPT-2; Radford et. al, 2019), and find a significant amount of offensive, factually unreliable, and otherwise toxic content. Our work provides a test bed for evaluating toxic generations by LMs and stresses the need for better data selection processes for pretraining.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, neutral): “Language models are capable of generating toxic language—including insults, hate speech, profanities and threats (Gehman et al., 2020; Rae et al., 2021).”
PaLM: Scaling Language Modeling with Pathways (2022, methodology): “We note that, for low TPP, the relatively high human TPC is due to the stratified sampling strategy (Gehman et al., 2020) used to create the underlying dataset.”
LLaMA: Open and Efficient Foundation Language Models (2023, background): “, 2019), and to generate toxic or offensive content (Gehman et al., 2020).”
Training language models to follow instructions with human feedback (2022, methodology): “To measure toxicity, we use the RealToxicityPrompts dataset (Gehman et al., 2020) and conduct both automatic and human evaluations.”
LaMDA: Language Models for Dialog Applications (2022, background): “[85], we find fine-tuning effective to augment language models for safety.”
Improving language models by retrieving from trillions of tokens (2021, background): “Large language models are prone to generating toxic outputs, as shown in Gehman et al. (2020).”
Challenges in Detoxifying Language Models (2021, methodology): “Prior work has considered various approaches towards reducing LM toxicity, either by fine-tuning a pre-trained LM (Gehman et al., 2020; Gururangan et al., 2020),”
Detoxifying Language Models Risks Marginalizing Minority Voices (2021, methodology): “Existing detoxiﬁca-tion strategies involve techniques such as ﬁnetun-ing LMs on nontoxic data (Gehman et al., 2020) or incorporating a toxicity discriminator during decoding (Dathathri et al., 2020).”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “Gehman et al. (2020) provide an in-depth analysis with respect to toxicity and fake news of O PEN W EB T EXT .”
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets (2021, background): “…sentence templates where terms pertaining to groups or their attributes are perturbed (Rudinger et al., 2018; Zhao et al., 2018; Kurita et al., 2019), prompts designed to elicit problematic responses (Gehman et al., 2020; Sheng et al., 2019; Example Sentences Context I really like Norweigan salmon.”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, background): “, 2022), generation of toxic language (Gehman et al., 2020) and enforcing stereotypes.”
Emergent Abilities of Large Language Models (2022, neutral): “(2021) found that while larger language models could produce more toxic responses from the RealToxicityPrompts dataset (Gehman et al., 2020), this behavior could be mitigated by giving models prompts with examples of being “helpful, harmless, and honest.”
OPT: Open Pre-trained Transformer Language Models (2022, methodology): “As shown in Section 4, we also ﬁnd OPT-175B has a high propensity to generate toxic language and reinforce harmful stereotypes, even when provided with a relatively innocuous prompt (Gehman et al., 2020), and adversarial prompts are trivial to ﬁnd (Dinan et al., 2021).”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “1 Hate, Toxicity, and Targeted Violence Beyond hate speech and toxic language, generations may also produce harmful biases [87], stereotypes [165] (overlapping with 4.”
The False Promise of Imitating Proprietary LLMs (2023, neutral): “In particular, we show in Figure 5 that as we finetune on more imitation data, the imitation model’s outputs become less toxic on RealToxicityPrompts (Gehman et al., 2020), as the model learns to abstain in a similar fashion to ChatGPT.”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “Gehman et al. (2020) built RealToxicityPrompts upon the Open-WebText corpus (Gokaslan and Cohen, 2019), containing 100k toxic prompts to test the toxic degeneration in GPT-2 (Radford et al., 2019).”
Pretraining Language Models with Human Preferences (2023, background): “Prompt pool For toxicity, we bootstrap the prompt pool with prompts from the “challenging” subset of RealToxicityPrompts (Gehman et al., 2020).”
The Capacity for Moral Self-Correction in Large Language Models (2023, background): “Large language models exhibit harmful social biases [1, 6, 8, 11, 15, 24, 29, 50, 62] that can sometimes get worse for larger models [2, 18, 20, 43, 55].”