Investigating Gender Bias in BERT

By Rishabh Bhardwaj, Navonil Majumder, Soujanya Poria. 2020

125 citations recorded by the source corpus.

In this work, we analyze the gender bias induced by BERT in downstream tasks. We also propose solutions to reduce gender bias. Contextual language models (CLMs) have pushed the NLP benchmarks to a new height. It has become a new norm to utilize CLM-provided word embeddings in downstream tasks such as text classification. However, unless addressed, CLMs are prone to learn intrinsic gender bias in the dataset. As a result, predictions of downstream NLP models can vary noticeably by varying gender words, such as replacing “he” to “she”, or even gender-neutral words. In this paper, we focus our analysis on a popular CLM, i.e., BERT\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {BERT}$$\end{document}. We analyze the gender bias it induces in five downstream tasks related to emotion and sentiment intensity prediction. For each task, we train a simple regressor utilizing BERT\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {BERT}$$\end{document}’s word embeddings. We then evaluate the gender bias in regressors using an equity evaluation corpus. Ideally and from the specific design, the models should discard gender informative features from the input. However, the results show a significant dependence of the system’s predictions on gender-particular words and phrases. We claim that such biases can be reduced by removing gender-specific features from word embedding. Hence, for each layer in BERT, we identify directions that primarily encode gender information. The space formed by such directions is referred to as the gender subspace in the semantic space of word embeddings. We propose an algorithm that finds fine-grained gender directions, i.e., one primary direction for each BERT layer. This obviates the need of realizing gender subspace in multiple dimensions and prevents other crucial information from being omitted. Experiments show that removing embedding components in gender directions achieves great success in reducing BERT-induced bias in the downstream tasks. The investigation reveals significant gender bias a contextualized language model ( i.e., BERT\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\text {BERT}$$\end{document}) induces in downstream tasks. The proposed solution seems promising in reducing such biases.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “Even though such language models have initially shown promising results, more recent investigation shows they may rely on various biases in their predictions (Hendricks et al., 2018; Brown et al., 2020; Bhardwaj et al., 2020; Kurita et al., 2019).”
What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus (2021, background): “…larger corpora generally result in better models (Kaplan et al., 2020; Sun et al., 2017), data quality and corpora content also plays a major role in the caliber and appropriateness of these models for the various downstream applications (Florez, 2019; Abid et al., 2021; Bhardwaj et al., 2021).”
Quantifying Societal Bias Amplification in Image Captioning (2022, methodology): “For completeness, we also report LIC when using BERT [15], although it has also been shown to exhibit bias [3,4] and it can affect our metric.”
Beneath the Tip of the Iceberg: Current Challenges and New Directions in Sentiment Analysis Research (2020, result): “Similarly, sentiment expression might vary across races and genders, e.g., as shown in a recent study by Bhardwaj et al.[9].”
A Study of Implicit Bias in Pretrained Language Models against People with Disabilities (2022, background): “Prior work identifying bias in NLP models has shown how they can be discriminatory against specific races (Mozafari et al., 2020; Ousidhoum et al., 2021) and genders (Pak and Paroubek, 2010; Bhardwaj et al., 2021).”
A Survey on Gender Bias in Natural Language Processing (2021, methodology): “The EEC dataset has been widely used in other projects [Bhardwaj et al. 2021] and has been extended with German sentences by Bartl et al. [2020].”
On Transferability of Bias Mitigation Effects in Language Model Fine-Tuning (2021, methodology): “Though previous work has addressed biases in frozen PTLM or word embeddings (Bolukbasi et al., 2016; Zhou et al., 2019; Bhardwaj et al., 2020; Liang et al., 2020; Ravfogel et al., 2020), for example by measuring associations between gender and occupations in an embedding space, they do not study…”