Challenges in Detoxifying Language Models

By Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John F. J. Mellor, Lisa Anne Hendricks, Kirsty Anderson, Pushmeet Kohli, Ben Coppin, Po-Sen Huang. 2021

248 citations recorded by the source corpus.

Large language models (LM) generate remarkably fluent text and can be efficiently adapted across NLP tasks. Measuring and guaranteeing the quality of generated text in terms of safety is imperative for deploying LMs in the real world; to this end, prior work often relies on automatic evaluation of LM toxicity. We critically discuss this approach, evaluate several toxicity mitigation strategies with respect to both automatic and human evaluation, and analyze consequences of toxicity mitigation in terms of model bias and LM quality. We demonstrate that while basic intervention strategies can effectively optimize previously established automatic metrics on the RealToxicityPrompts dataset, this comes at the cost of reduced LM coverage for both texts about, and dialects of, marginalized groups. Additionally, we find that human raters often disagree with high automatic toxicity scores after strong toxicity reduction interventions -- highlighting further the nuances involved in careful evaluation of LM toxicity.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, neutral): “While toxicity is an umbrella term, and its evaluation in LMs comes with challenges (Welbl et al., 2021; Xu et al., 2021), automatic classiﬁer scores can provide an indication for the levels of harmful text that a LM generates.”
PaLM: Scaling Language Modeling with Pathways (2022, methodology): “To evaluate toxicity degeneration, we adapt the methodology used in Welbl et al. (2021); Rae et al. (2021).”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “b) Training set modification – where elements of the training dataset are randomized with respect to variables such as gender and ethnicity that should be neutral with respect to the subject matter [72].”
Ethical and social risks of harm from Language Models (2021, methodology): “Great strides have been made in developing risk mitigation tools, including by (Chen et al., 2021a; Dinan et al., 2021; Solaiman and Dennison, 2021; Welbl et al., 2021) and others mentioned in the above taxonomy.”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “…was motivated by: (1) challenges in avoiding overfiltering content from minorities when using ML-based classifiers on the content of documents (Welbl et al., 2021); (2) NSFW words block-list applied on content (such as the one used in C4) also resulting in overfiltering of legal and medical…”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “As the first concern, toxicity and unfair contents have been extensively studied, and relevant research like toxicity detection, detoxification, and bias mitigation have made considerable progress (Schmidt and Wiegand, 2017; Gehman et al., 2020; Welbl et al., 2021).”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, methodology): “• Well-being Survey : Similar to [58], we distributed a survey to measure the effects of, and worker feelings towards, the Review Task.”
A General Language Assistant as a Laboratory for Alignment (2021, background): “…to reduce toxicity (e.g., ﬁne-tuning models on non-toxic training data, steering/ﬁltering model outputs away from toxic outputs at test time, ﬁltering toxic training data at train time) can yield much larger decreases in automated toxicity scores than the ones we observe here [GGS + 20, WGU + 21].”
Red Teaming Language Models with Language Models (2022, background): “Harmful text classifiers are often inaccurate or biased (Gehman et al., 2020; Welbl et al., 2021).”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “, 2021), supervised fine-tuning on curated data (Solaiman & Dennison, 2021; Ngo et al., 2021; Welbl et al., 2021; Chung et al., 2022) or selected samples from the model (Zelikman et al.”
On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgetting (2022, background): “Fine-tuning large language models has become an active area of research, due to its importance in adapting large language models to satisfy task-level preferences, or in combating their social risks such as “distributional” stereotyping (Weidinger et al., 2021; Welbl et al., 2021).”
Predictability and Surprise in Large Generative Models (2022, background): “Different people often have different perceptions about text with the same toxicity score [73].”
Multimodal datasets: misogyny, pornography, and malignant stereotypes (2021, background): “On the other hand, as recently demonstrated in studies such as [60, 61], granular safe ﬁltering of the datasets created and the downstream detoxiﬁcation of the models trained on such datasets remain a tenuous and laborious work.”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, methodology): “In LM Benchmarks: In line with this, the Perspective API is used to identify LM outputs or training data documents which are toxic [40, 103, 106, 21, 85].”
Director: Generator-Classifiers For Supervised Language Modeling (2022, neutral): “Biases may be ampliﬁed from those already present in the large training corpora, and toxic or otherwise unsafe language can be generated (Gehman et al., 2020; Welbl et al., 2021).”
AlexaTM 20B: Few-Shot Learning Using a Large-Scale Multilingual Seq2Seq Model (2022, methodology): “Following PaLM (Chowdhery et al., 2022) which borrows the setup from (Welbl et al., 2021) and (Rae et al., 2021), we randomly sample 10K prompts from RealToxicityPrompts dataset (Gehman et al., 2020) and generate 25 continuations for each prompt with top-k sampling with k=40 and a temperature of…”
GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (2023, background): “They can produce toxic or mis-leading content [15,21,66] and are susceptible to “hallucina-tions” that result in nonsensical or untruthful outputs [33,45].”