Identifying and Reducing Gender Bias in Word-Level Language Models

By Shikha Bordia, Samuel R. Bowman. 2019

372 citations recorded by the source corpus.

Many text corpora exhibit socially problematic biases, which can be propagated or amplified in the models trained on such data. For example, doctor cooccurs more frequently with male pronouns than female pronouns. In this study we (i) propose a metric to measure gender bias; (ii) measure bias in a text corpus and the text generated from a recurrent neural network language model trained on the text corpus; (iii) propose a regularization loss term for the language model that minimizes the projection of encoder-trained embeddings onto an embedding subspace that encodes gender; (iv) finally, evaluate efficacy of our proposed method on reducing gender bias. We find this regularization method to be effective in reducing gender bias up to an optimal weight assigned to the loss term, beyond which the model becomes unstable as the perplexity increases. We replicate this study on three training corpora—Penn Treebank, WikiText-2, and CNN/Daily Mail—resulting in similar conclusions.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “…et al. (2017); Dixon et al. (2018); Kiritchenko and Mohammad (2018); Shen et al. (2018); Zhao et al. (2018b); Bhaskaran and Bhallamudi (2019); Bordia and Bowman (2019); Brunet et al. (2019); Chaloner and Maldonado (2019); De-Arteaga et al. (2019); Dev and Phillips (2019); Font and…”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “Previous work (Sheng et al., 2019; Bordia and Bowman, 2019; Nadeem et al., 2021; de Vassimon Manela et al., 2021) has observed language models absorb bias and toxicity represented in the training data.”
Societal Biases in Language Generation: Progress and Challenges (2021, methodology): “Language models are the core components for many NLG and NLU tasks, and this task enables directly quantifying biases in large, pre-trained language models (Bordia and Bowman, 2019; Sheng et al., 2019; Solaiman et al., 2019; Brown et al., 2020).”
Persistent Anti-Muslim Bias in Large Language Models (2021, background): “Previous work has explored the biases that are ultimately learned by language models, with the primary focus being racial and gender stereotypes [2, 3, 7, 12].”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, methodology): “tors to measure social bias began as a technique specific for probing the gender associations of static word embeddings (Bolukbasi et al., 2016; Caliskan et al., 2017; Bordia and Bowman, 2019).”
BBQ: A hand-built bias benchmark for question answering (2021, background): “Measuring Bias in NLP Several studies have investigated the prevalence of biases in NLP models (Caliskan et al., 2017; May et al., 2019; Bordia and Bowman, 2019; Davidson et al., 2019; Magee et al., 2021), with many focusing on cases of models exhibiting stereotyping behavior.”
GeDi: Generative Discriminator Guided Sequence Generation (2020, background): “Other work has focused on removing gender bias from language models (Bordia and Bowman, 2019; Dinan et al., 2020; Bolukbasi et al., 2016).”
Does Gender Matter? Towards Fairness in Dialogue Systems (2019, methodology): “In the work (Bordia and Bowman, 2019), a measurement is introduced for measuring gender bias in a text generated from a language model that is trained on a text corpus along with measuring the bias in the training text itself.”
Memory-assisted prompt editing to improve GPT-3 after deployment (2022, background): “Finally, large-language models generate text that might be biased and insensitive to a user’s socio-cultural context (Bordia and Bowman, 2019; Sharma et al., 2021; Hovy and Prabhumoye, 2021).”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “…models (Gira et al., 2022; Kirtane et al., 2022; Choenni et al., 2021), or the relationship between the corpus statistics and the measured bias (Bordia & Bowman, 2019; Van der Wal et al., 2022b), researchers have generally lacked the tools to study the role of the training data on the…”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, background): “Imagen also relies on text encoders trained on uncurated web-scale data, and thus inherits the social biases and limitations of large language models [5, 3, 52].”
Modifying Memories in Transformer Models (2020, background): “Furthermore, language models are susceptible to various biases present in the large corpora of text used for their training, and such biases may need to be eliminated to ensure a fair application of such models in real-world (Bolukbasi et al., 2016; Bordia and Bowman, 2019; Blodgett et al., 2020).”
A Survey on Bias and Fairness in Machine Learning (2019, background): “In [20] authors introduce a metric for measuring gender bias in a generated text from a language model based on recurrent neural networks that is trained on a text corpus along with measuring the bias in the training text itself.”
A Survey on Bias and Fairness in Natural Language Processing (2022, neutral): “Bordia and Bowman (2019) deﬁne the probability of a word occuring in context with gen-dered words as where c ( w, g ) is the count of the word in a speciﬁed context window and g is the set of gendered words which belong to the male ( m ) or the female ( f ) set.”