Reducing Sentiment Bias in Language Models via Counterfactual Evaluation

By Po-Sen Huang, Huan Zhang, Ray Jiang, Robert Stanforth, Johannes Welbl, Jack W. Rae, Vishal Maini, Dani Yogatama, Pushmeet Kohli. 2019

251 citations recorded by the source corpus.

Advances in language modeling architectures and the availability of large text corpora have driven progress in automatic text generation. While this results in models capable of generating coherent texts, it also prompts models to internalize social biases present in the training corpus. This paper aims to quantify and reduce a particular type of bias exhibited by language models: bias in the sentiment of generated text. Given a conditioning context (e.g., a writing prompt) and a language model, we analyze if (and how) the sentiment of the generated text is affected by changes in values of sensitive attributes (e.g., country names, occupations, genders) in the conditioning context using a form of counterfactual evaluation. We quantify sentiment bias by adopting individual and group fairness metrics from the fair machine learning literature, and demonstrate that large-scale models trained on two different corpora (news articles, and Wikipedia) exhibit considerable levels of bias. We then propose embedding and sentiment prediction-derived regularization on the language model’s latent representations. The regularizations improve fairness metrics while retaining comparable levels of perplexity and semantic similarity.

Subfields: Artificial Intelligence, General Social Sciences, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language Models are Few-Shot Learners (2020, background): “The literature on this is also extensive [QMZH19, HZJ + 19], so we offer only a few brief comments on future directions speciﬁc to large language models.”
Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “…Davidson et al. (2019); Dev et al. (2019); Dinan et al. (2019); Fisher (2019); Florez (2019); Font and Costa-jussà (2019); Garg et al. (2019); Huang et al. (2019); Liu et al. (2019); Nozza et al. (2019); Prabhakaran et al. (2019); Qian et al. (2019); Sap et al. (2019); Stanovsky et al.…”
Challenges in Detoxifying Language Models (2021, background): “…typically does not capture less obvious, but no less important harms—such as subtle or distributional biases (Sap et al., 2019b; Sheng et al., 2019; Huang et al., 2020; Abid et al., 2021). by steering a model’s generation towards text less likely to be classified as toxic (Dathathri et al., 2020;…”
Societal Biases in Language Generation: Progress and Challenges (2021, methodology): “…generations, Ma et al. (2020) increasing the accuracy of the change in agency, Zmigrod et al. (2019) increasing the number of correct inflections, Huang et al. (2020) reducing individual and group fairness scores, and Sheng et al. (2021b) reducing the amount of ad hominems towards marginalized…”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “…measurement templates, which form the basis for prompts that can be used to measure bias (Rudinger et al., 2018; May et al., 2019; Sheng et al., 2019; Kurita et al., 2019; Webster et al., 2020; Gehman et al., 2020; Huang et al., 2020; Vig et al., 2020; Kirk et al., 2021; Perez et al., 2022).”
Alignment of Language Agents (2021, background): “Another line of work aims to not only measure but also mitigate the harmful content from an LLM. Huang et al. (2019) introduce a general framework to reduce bias under a certain measure (e.g. sentiment) for text generated by a language model, given sensitive attributes.”
Red Teaming Language Models with Language Models (2022, background): “Prior work discovers such biases against certain groups by hypothesizing discriminated-against groups in advance (e.g., a particular gender, race, or religion; Bolukbasi et al., 2016; Sheng et al., 2019; Huang et al., 2020).”
CausaLM: Causal Model Explanation Through Counterfactual Language Models (2020, methodology): “Model debiasing is done through manipulation of the training data (Kaushik, Hovy, and Lipton 2020), by altering the training process (Huang et al. 2020), or by changing the model (Gehrmann et al.”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “…change the training data (e.g., Lu et al., 2018; Hall Maudslay et al., 2019; Zhao et al., 2018a; Kaushik et al., 2019), the training process (e.g., Huang et al., 2019; Qian et al., 2019), or the model itself (e.g., Madras et al., 2019; Romanov et al., 2019; Gehrmann et al., 2019) to ensure…”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, methodology): “In line with this idea, Huang et al. (2020) used a sentiment classiﬁer to quan-278 tify and reduce the sentiment bias existent in LLMs. Similarly, the CrowS-Pairs benchmark (Nangia et al., 2020) leverages the paradigm of minimal pairs to con-281 trast sentences expressing stereotypes against social…”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “…BAD[107] None* BOLD [37] Profession, Gender, Race, Religion, Political Ideology Stereoset [78] Gender, Profession, Race, Religion Sentiment Bias [54, 21, 85] Race, Country, Religion, Gender, Profession BBQ [83] Age, Disability Statues, Gender Identity, Nationality, Physical Appearance, Race /…”
Taxonomy of Risks posed by Language Models (2022, background): “Counterfactual evaluation [82] showed that the LM Gopher associates negative sentiment with different social groups [148].”
Investigating Societal Biases in a Poetry Composition System (2020, background): “, 2018; Kiritchenko and Mohammad, 2018), and language modeling (Sheng et al., 2019; Bordia and Bowman, 2019; Pryzant et al., 2020; Huang et al., 2019).”
Biases in Large Language Models: Origins, Inventory, and Discussion (2023, result): “Pretrained language models are unmistakably and, sometimes, blatantly, biased in several respects, as numerous studies have shown over the years [1, 3, 10, 21, 67, 77, 94].”