Gender Bias in Contextualized Word Embeddings

By Jieyu Zhao, Tianlu Wang, Mark Yatskar, Ryan Cotterell, Vicente Ordonez, Kai-Wei Chang. 2019

473 citations recorded by the source corpus.

In this paper, we quantify, analyze and mitigate gender bias exhibited in ELMo’s contextualized word vectors. First, we conduct several intrinsic analyses and find that (1) training data for ELMo contains significantly more male than female entities, (2) the trained ELMo embeddings systematically encode gender information and (3) ELMo unequally encodes gender information about male and female entities. Then, we show that a state-of-the-art coreference system that depends on ELMo inherits its bias and demonstrates significant bias on the WinoBias probing corpus. Finally, we explore two methods to mitigate such gender bias and show that the bias demonstrated on WinoBias can be eliminated.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “We found that nearly all of the papers focus on system predictions as the potential sources of “bias,” with many additionally focusing on “bias” in datasets (e.g., differences in the number of gendered pronouns in the training data (Zhao et al., 2019)).”
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, background): “However, recent studies have found that both context-free and context-dependent word embedding models contain human-like semantic biases, including gender and race (Bolukbasi et al., 2016; Caliskan et al., 2017; Zhao et al., 2019).”
Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor (2019, methodology): “5 Recent work also suggests that, compared to word embeddings, contextualised embeddings (Devlin et al. 2018; Peters et al. 2018) might be less biased (Zhao et al. 2019; Basta, Costa-jussà, and Casas 2019).”
Ethical and social risks of harm from Language Models (2021, neutral): “languages (Nozza et al., 2021), and discriminatory gender biases were found in contextual word embedding by BERT (Kurita et al., 2019) and ELMo (Zhao et al., 2019).”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “A wealth of work has shown that toxicity and social biases in training data are acquired by large pretrained sentence encoders (e.g., gender bias in BERT; May et al., 2019; Zhao et al., 2019; Basta et al., 2019; Kurita et al., 2019).”
Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies (2021, neutral): “There have been multiple attempts to understand social biases in language processing (Sheng et al., 2021; Caliskan et al., 2017), quantify them (Rudinger et al., 2018; Webster et al., 2018; De-Arteaga et al., 2019), and mitigate them (Zhao et al., 2019; Ravfo-gel et al., 2020; Sun et al., 2019).”
On Measuring and Mitigating Biased Inferences of Word Embeddings (2019, methodology): “Coreference resolution, especially pronoun coreference, has been recently used as an extrinsic probe to measure bias in representations (e.g., (Rudinger et al. 2018), (Zhao et al. 2019), (Webster et al. 2018)).”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Popular evaluations for biases use association tests [46] or examine stereotypes [157, 156, 138], correlations and co-occurrences [272], and sentiment analysis [66].”
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, background): “Previous work have leveraged power dynamics between two groups to quantify representational harms (Zhao et al., 2018; Rudinger et al., 2018; Zhao et al., 2019; Vig et al., 2020; Nadeem et al., 2021; Nangia et al., 2020).”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “Some even select existing sentences from text sources and swap demographic terms heuristically (Zhao et al., 2019; Ma et al., 2021; Wang et al., 2021; Papakipos and Bitton, 2022), utilize handcrafted grammars (Renduchintala et al., 2021), or use machine-learned systems to swap descriptors (Qian et…”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, background): “Gender-occupation associations are also discernible from distributional word statistics (Bolukbasi et al., 2016; Caliskan et al., 2017; Basta et al., 2019; Zhao et al., 2019).”
Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models (2021, methodology): “To expose this, Zhao et al. (2019) analyse the behaviour of a coreference resolution system proposed by Lee et al. (2018) with ELMo contextual weights on the WinoBias benchmark, revealing a signiﬁcant disparity in performance on the pro-and anti-stereotypical datasets.”
Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, background): “Particular attention has been paid to uncovering, analyzing, and removing gender biases in word embeddings (Basta et al., 2019; Kaneko and Bollegala, 2019; Zhao et al., 2019, 2018b; Bolukbasi et al., 2016).”
Measuring Bias in Contextualized Word Representations (2019, background): “For instance, Zhao et al. (2019) showed that ELMo exhibits gender bias for certain pro-fessions.”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “Gender bias has surfaced as a major concern in word representations, both static word embeddings (Caliskan et al., 2017; Bolukbasi et al., 2016) and contextualized word representations (Zhao et al., 2019a; Basta et al., 2019; Tan and Celis, 2019).”
Mitigating Gender Bias in Natural Language Processing: Literature Review (2019, methodology): “…trained on OntoNotes 5.0 which was tested on WinoBias, gender augmentation lowered the difference be-tween F 1 scores on pro-stereotypical and anti-stereotypical test sets signiﬁcantly, which indicates the model was less inclined to make gender-biased predictions (Zhao et al., 2018a, 2019).”
Evaluating the Underlying Gender Bias in Contextualized Word Embeddings (2019, background): “A similar conclusion was stated in the recent work (Zhao et al., 2019) where the authors applied the same approach, but for gender swapped variants of sentences with professions.”
Assessing Social and Intersectional Biases in Contextualized Word Representations (2019, methodology): “[35] in counting, in these datasets, the occurrence of a male pronoun or a female pronoun in a sentence, as well as the co-occurrences of stereotypically gendered occupation words with the respective pronouns in a sentence.”