Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP

By Timo Schick, Sahana Udupa, Hinrich Schütze. 2021

496 citations recorded by the source corpus.

Abstract ⚠ This paper contains prompts and model outputs that are offensive in nature. When trained on large, unfiltered crawls from the Internet, language models pick up and reproduce all kinds of undesirable biases that can be found in the data: They often generate racist, sexist, violent, or otherwise toxic language. As large models require millions of training examples to achieve good performance, it is difficult to completely prevent them from being exposed to such content. In this paper, we first demonstrate a surprising finding: Pretrained language models recognize, to a considerable degree, their undesirable biases and the toxicity of the content they produce. We refer to this capability as self-diagnosis. Based on this finding, we then propose a decoding algorithm that, given only a textual description of the undesired behavior, reduces the probability of a language model producing problematic text. We refer to this approach as self-debiasing. Self-debiasing does not rely on manually curated word lists, nor does it require any training data or changes to the model’s parameters. While we by no means eliminate the issue of language models generating biased text, we believe our approach to be an important step in this direction.1

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “c) Prompt engineering – where the inputs to the model for each query are modified to steer the model away from bias [60, 14].”
Challenges in Detoxifying Language Models (2021, background): “This typically relies on an external toxicity classifier, although Schick et al. (2021) show that even a LM’s own toxicity self-diagnosis can be used to this end.”
Societal Biases in Language Generation: Progress and Challenges (2021, background): “…et al., 2019; Sheng et al., 2019, 2020; Shwartz et al., 2020; Vig et al., 2020; Yeo and Chen, 2020; Huang et al., 2020; Dhamala et al., 2021; Schick et al., 2021), GPT-3 (Brown et al., 2020), CTRL (Dhamala et al., 2021), TransformerXL (Shwartz et al., 2020; Vig et al., 2020; Huang et al.,…”
OPT: Open Pre-trained Transformer Language Models (2022, neutral): “…for toxicity and biases (Dathathri et al., 2019; Dinan et al., 2019a; Sheng et al., 2019; Dinan et al., 2020a; Liu et al., 2019a; Krause et al., 2020; Xu et al., 2020; Liang et al., 2021; Dinan et al., 2021; Xu et al., 2021a; Dhamala et al., 2021; Schick et al., 2021; Ouyang et al., 2022).”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “Schick et al. (2021) discovered that LMs are able to identify the harmful responses generated by themselves, whereby they proposed a decoding algorithm named self-debiasing by giving undesired text descriptions.”
The Capacity for Moral Self-Correction in Large Language Models (2023, background): “Our hypothesis is not entirely new (see §2 for related work, especially [51, 63]) but we believe our experiments and results are.”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Controlled Decoding Toxicity Autoregressive toxic content control (Yang and Klein, 2021; Liu et al., 2021a; Dathathri et al., 2019; Krause et al., 2021; Schick et al., 2021; Lu et al., 2021; Pascual et al., 2021; Wolf et al., 2020); Non-autoregressive toxic content control(Kumar et al.”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “Some bias measurement approaches, such as selfdebiasing (Schick et al., 2021), do not require a list of terms at all.”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “In the context of safety, Schick et al. (2021) ﬁnd that LLMs show an ability to self-identify and mitigate toxic generations using prompt manipulation.”
Guiding the Release of Safer E2E Conversational AI through Value Sensitive Design (2022, background): “In addition, work on building safer LLMs explores fine-tuning on curated data (Solaimon and Dennison, 2021) or directly controlling the generations of the model (Dathathri et al., 2019; Liu et al., 2021; Schick et al., 2021; Xu et al., 2020).”
Mitigating harm in language models with conditional-likelihood filtration (2021, positive): “Self-debiasing [Schick et al., 2021] mitigates corpus-based bias in language models during generation by using the learned knowledge of a language model to identify biases in text with a zero-shot prompt-based approach.”
Red Teaming Language Models with Language Models (2022, background): “Lastly, debiasing LMs is an active area (Huang et al., 2020; Brown et al., 2020; Xu et al., 2021a; Schick et al., 2021, inter alia), where progress will improve LM-based red teaming.”
Generating Datasets with Pretrained Language Models (2021, methodology): “We achieve this using Schick et al. (2021)’s self-debiasing algorithm: When sampling the token xk, we consider not just py = pM (xk | Iy(x1), x1, . . . , xk−1) [xk’s probability given Iy(x1)], but also py′ [xk’s probability given Iy′(x1)], for all y′ ∈ CL(y).”
Constructing Highly Inductive Contexts for Dialogue Safety through Controllable Reverse Generation (2022, background): “Some works employed hard or soft prompts (Keskar et al., 2019; Schick et al., 2021), while others explicitly change the token distribution during generation (Dathathri et al.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “Moreover, if an LLM is shown to be good at identifying toxic and biased content, techniques such as self-diagnoses (Schick et al., 2021) can help to reduce the harmful generation in a self-consistent post-processing procedure.”
Using In-Context Learning to Improve Dialogue Safety (2023, methodology): “Following previous work (Schick et al., 2021; Lu et al., 2022a), we use a threshold of 0 .”
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus (2021, methodology): “, 2020), and develop methods for evaluating the bias in trained models (Schick et al., 2021).”