Detoxifying Language Models Risks Marginalizing Minority Voices

By Albert Xu, Eshaan Pathak, Eric Wallace, Suchin Gururangan, Maarten Sap, Daniel Klein. 2021

148 citations recorded by the source corpus.

Language models (LMs) must be both safe and equitable to be responsibly deployed in practice. With safety in mind, numerous detoxification techniques (e.g., Dathathri et al. 2020; Krause et al. 2020) have been proposed to mitigate toxic LM generations. In this work, we show that these detoxification techniques hurt equity: they decrease the utility of LMs on language used by marginalized groups (e.g., African-American English and minority identity mentions). In particular, we perform automatic and human evaluations of text generation quality when LMs are conditioned on inputs with different dialects and group identifiers. We find that detoxification makes LMs more brittle to distribution shift, especially on language used by marginalized groups. We identify that these failures stem from detoxification methods exploiting spurious correlations in toxicity datasets. Overall, our results highlight the tension between the controllability and distributional robustness of LMs.

Subfields: Artificial Intelligence, General Health Professions, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Health Professions

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, neutral): “While toxicity is an umbrella term, and its evaluation in LMs comes with challenges (Welbl et al., 2021; Xu et al., 2021), automatic classiﬁer scores can provide an indication for the levels of harmful text that a LM generates.”
Ethical and social risks of harm from Language Models (2021, methodology): “For example, methods to reduce toxic speech from LMs have been found to bias model prediction against marginalised groups (Welbl et al., 2021; Xu et al., 2021).”
Challenges in Detoxifying Language Models (2021, background): “Likelihood-based metrics are ubiquitous within language modeling in general, as well for evaluating biases both in other work (Xu et al., 2021) and our own.”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “However, the removal of harmful content can come with its own costs of lower general performances for sub-populations that use models for generation [269].”
Scaling Instruction-Finetuned Language Models (2022, background): “Automated measures of toxic language contain several forms of noise and bias (Xu et al., 2021; Garg et al., 2022), and do not consider diverse perspectives on toxic language (Goyal et al.”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “Xu et al. (2021a) show that popular methods for mitigating toxic generation in LLMs decreases the utility of these models on marginalized groups.”
Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets (2021, neutral): “However, [38] found that technical detoxification methods”
Red Teaming Language Models with Language Models (2022, methodology): “We subsample n = 2 dialogues uniformly at random from BAD dataset dialogues with 7+ offensive chatbot replies (85 dialogues total).”
Multimodal datasets: misogyny, pornography, and malignant stereotypes (2021, background): “On the other hand, as recently demonstrated in studies such as [60, 61], granular safe ﬁltering of the datasets created and the downstream detoxiﬁcation of the models trained on such datasets remain a tenuous and laborious work.”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, methodology): “In LM Benchmarks: In line with this, the Perspective API is used to identify LM outputs or training data documents which are toxic [40, 103, 106, 21, 85].”
Taxonomy of Risks posed by Language Models (2022, background): “Importantly, mitigation approaches are likely to work best if they take a broad perspective of the overall risk landscape and occur in concert, to avoid addressing one risk in a way that aggravates another [194, 197].”
SafetyKit: First Aid for Measuring Safety in Open-domain Conversational Systems (2022, background): “Xu et al. (2021a) show that popular methods for mitigating toxic generation in LLMs decreases the utility of these models on marginalized groups, potentially resulting in harms such as forcing marginalized users to code-switch.”
Mix and Match: Learning-free Controllable Text Generationusing Energy Language Models (2022, methodology): “This approach (Yang and Klein, 2021) 1015 trains step-wise discriminators on partial gen- 1016 erations from GPT-2 to determine whether the 1017 constraints related to desired attributes will be 1018 satisfied by the future completion of the sequence 1019 or not.”
Toxicity in Multilingual Machine Translation at Scale (2022, background): “One advantage of this type of classifier is transparency, which diminishes the possibility of covering up biases (Xu et al., 2021).”