Towards Understanding and Mitigating Social Biases in Language Models

By Percy Liang, Chiyu Wu, Louis-philippe Morency, Ruslan Salakhutdinov. 2021

508 citations recorded by the source corpus.

As machine learning methods are deployed in real-world settings such as healthcare, legal systems, and social science, it is crucial to recognize how they shape social biases and stereotypes in these sensitive decision-making processes. Among such real-world deployments are large-scale pretrained language models (LMs) that can be potentially dangerous in manifesting undesirable representational biases - harmful biases resulting from stereotyping that propagate negative generalizations involving gender, race, religion, and other social constructs. As a step towards improving the fairness of LMs, we carefully define several sources of representational biases before proposing new benchmarks and metrics to measure them. With these tools, we propose steps towards mitigating social biases during text generation. Our empirical results and human evaluation demonstrate effectiveness in mitigating bias while retaining crucial contextual information for high-fidelity text generation, thereby pushing forward the performance-fairness Pareto frontier.

Subfields: Artificial Intelligence, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Training language models to follow instructions with human feedback (2022, background): “Language models can produce biased outputs (Dhamala et al., 2021; Liang et al., 2021; Manela et al., 2021; Caliskan et al., 2017; Kirk et al., 2021), leak private data (Carlini et al., 2021), generate misinformation (Solaiman et al., 2019; Buchanan et al., 2021), and be used maliciously; for a…”
OPT: Open Pre-trained Transformer Language Models (2022, background): “There has been a great deal of work on mitigations for toxicity and biases (Dathathri et al., 2019; Dinan et al., 2019a; Sheng et al., 2019; Dinan et al., 2020a; Liu et al., 2019a; Krause et al., 2020; Xu et al., 2020; Liang et al., 2021; Dinan et al., 2021; Xu et al., 2021a; Dhamala et al., 2021; S”
The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, methodology): “In the absence of such methods, language models are known to demonstrate toxic/harmful behaviour (Sheng et al., 2019; Liang et al., 2021; Wallace et al., 2019), generate non-factual information (Maynez et al.”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, positive): “Evidence for this behavior has been substantially corroborated in NLP literature (Nadeem et al., 2021; Nozza et al., 2021; Huang et al., 2020b; Hassan et al., 2021; Blodgett et al., 2020; Liang et al., 2021; Field et al., 2021, inter alia).”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, background): “Many strategies have been developed to address some of these harms (e.g., [58, 4, 48, 36, 34, 19, 60]).”
RuleBERT: Teaching Soft Rules to Pre-Trained Language Models (2021, methodology): “Biases We are aware of (i) the biases and abusive language patterns (Sheng et al., 2019; Zhang et al., 2020; Bender et al., 2021; Liang et al., 2021) that PLMs impose, and (ii) the imperfectness and the biases of our rules as data from Wikipedia has been used to mine the rules and compute their…”
GLM-130B: An Open Bilingual Pre-trained Model (2022, background): “Also, it is known that LLMs can suffer from problems in fairness, bias, privacy, and truthfulness Zhang et al. (2021); Lin et al. (2022); Liang et al. (2021); Bender et al. (2021).”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, background): “…produce harmful content when addressing certain topics (Gehman et al., 2020; Lin et al., 2021) and demograph-028 ics (Sheng et al., 2019; Liang et al., 2021; Dev et al., 2021a)—in part due to the training data used (Dunn, 2020; Gao et al., 2020; Bender et al., 2021), and the design of…”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “…Winogender [87] Gender Winobias [108] Gender Gender & Occ [21, 85] Gender Deconfounding [43] Gender, Profession, Country TruthfulQA [74] None DTC [71] Gender, Religion Muslim Bias [5] Religion BAD[107] None* BOLD [37] Profession, Gender, Race, Religion, Political Ideology Stereoset [78] Gender,…”
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, background): “To leverage the formidable capabilities of LLMs in practical applications, a growing amount of research has underscored the importance of aligning these models with human preferences (Agrawal et al., 2023; Kadavath et al., 2022; Shi et al., 2023; Liang et al., 2021; Sheng et al., 2019).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “[300] proposed A-INLP approach which dynamically identifies bias-”
Towards Measuring the Representation of Subjective Global Opinions in Language Models (2023, unknown): “Several studies have found harmful biases related to gender, race, religion and other attributes in these models [71, 75, 1, 13, 56, 50, 48, 17].”
Gendered Mental Health Stigma in Masked Language Models (2022, background): “There is a large body of prior work on bias in NLP models, particularly focusing on gender, race, and disability (Garrido-Muñoz et al., 2021; Blodgett et al., 2020; Liang et al., 2021).”
Exploring AI Ethics of ChatGPT: A Diagnostic Analysis (2023, background): “Consequently, numerous research efforts have been undertaken on the AI ethics of LLMs, ranging from discovering unethical behavior to mitigating bias [14].”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, background): “Extensive research has been conducted on the potential harm caused by deployed models [229–231], which can include biased outputs [232, 233], stereotypes [234], and misinformation [235].”
On the Planning Abilities of Large Language Models - A Critical Investigation (2023, background): “There is significant literature on the social harms–such as the perpetuation of biases–caused by the text generated by LLMs as they are trained on unwashed web data [27, 18].”