Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models

By Daniel de Vassimon Manela, D. Errington, David Errington, Thomas Fisher, Boris van Breugel, Pasquale Minervini. 2021

102 citations recorded by the source corpus.

This paper proposes two intuitive metrics, skew and stereotype, that quantify and analyse the gender bias present in contextual language models when tackling the WinoBias pronoun resolution task. We find evidence that gender stereotype correlates approximately negatively with gender skew in out-of-the-box models, suggesting that there is a trade-off between these two forms of bias. We investigate two methods to mitigate bias. The first approach is an online method which is effective at removing skew at the expense of stereotype. The second, inspired by previous work on ELMo, involves the fine-tuning of BERT using an augmented gender-balanced dataset. We show that this reduces both skew and stereotype relative to its unaugmented fine-tuned counterpart. However, we find that existing gender bias benchmarks do not fully probe professional bias as pronoun resolution may be obfuscated by cross-correlations from other manifestations of gender prejudice.

Subfields: Artificial Intelligence, General Social Sciences, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Training language models to follow instructions with human feedback (2022, background): “Language models can produce biased outputs (Dhamala et al., 2021; Liang et al., 2021; Manela et al., 2021; Caliskan et al., 2017; Kirk et al., 2021), leak private data (Carlini et al., 2021), generate misinformation (Solaiman et al., 2019; Buchanan et al., 2021), and be used maliciously; for a…”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “Previous work (Sheng et al., 2019; Bordia and Bowman, 2019; Nadeem et al., 2021; de Vassimon Manela et al., 2021) has observed language models absorb bias and toxicity represented in the training data.”
Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias (2022, methodology): “Other works measured bias in language models directly using masked language modeling (MLM) (Nadeem et al., 2021; Nangia et al., 2020; de Vassimon Manela et al., 2021). ar X iv :2 20 6.”
A Survey on Gender Bias in Natural Language Processing (2021, methodology): “Then, the amount of bias encoded in NLP systems can be quantified using: accuracy (percentage of observations with the correctly gendered entity) [Saunders and Byrne 2020]; difference in accuracy between the set of sentences with anti-stereotypical and stereotypical sentences; 𝐹1 score and differen”
Improving Gender Fairness of Pre-Trained Language Models without Catastrophic Forgetting (2021, background): “Among various gender-biased behaviours of pre-trained models, bias on professions is the most prominent and well-studied (de Vassimon Manela et al., 2021; Vig et al., 2020; Qian et al., 2019; Zhao et al., 2019).”
Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you? (2021, unknown): “To the best of our knowledge, Nadeem et al. (2020); Nangia et al. (2020) and Manela et al. (2021) are the ﬁrst to explicitly study stereotypes in pretrained sentence encoders.”
Your fairness may vary: Pretrained language model fairness in toxic text classification (2021, background): “…several studies have documented the bias of LMs (Bolukbasi et al., 2016; Hutchinson et al., 2020; Webster et al., 2020; Borkan et al., 2019; de Vassimon Manela et al., 2021) and others discuss potential societal harms (Blodgett et al., 2020; Bender et al., 2021) for individuals or groups.”
Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study (2023, background): “Although the biases of pretrained language models (PLMs) have been extensively studied (de Vassimon Manela et al., 2021; Steed et al., 2022; Li et al., 2022), finding a consistent methodology for measuring their cultural alignment remains an open research question.”