Understanding Undesirable Word Embedding Associations

By Kawin Ethayarajh, David Kristjanson Duvenaud, Graeme Hirst. 2019

137 citations recorded by the source corpus.

Word embeddings are often criticized for capturing undesirable word associations such as gender stereotypes. However, methods for measuring and removing such biases remain poorly understood. We show that for any embedding model that implicitly does matrix factorization, debiasing vectors post hoc using subspace projection (Bolukbasi et al., 2016) is, under certain conditions, equivalent to training on an unbiased corpus. We also prove that WEAT, the most common association test for word embeddings, systematically overestimates bias. Given that the subspace projection method is provably effective, we use it to derive a new measure of association called the relational inner product association (RIPA). Experiments with RIPA reveal that, on average, skipgram with negative sampling (SGNS) does not make most words any more gendered than they are in the training corpus. However, for gender-stereotyped words, SGNS actually amplifies the gender association in the corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “Surveys, frameworks, and meta-analyses Hovy and Spruit (2016); Larson (2017); McCurdy and Serbetçi (2017); Schnoebelen (2017); Basta et al. (2019); Ethayarajh et al. (2019); Gonen and Goldberg (2019); Lauscher and Glavaš (2019); Loukina et al. (2019); Mayfield et al. (2019); Mirzaev et al. (2019);…”
Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection (2020, positive): “This observation aligns with the analysis of Ethayarajh et al. (2019) who demonstrated that debiasing by projection is theoretically effective provided that one removes all relevant directions in the latent space.”
Examining Gender Bias in Languages with Grammatical Gender (2019, background): “Therefore, extensive effort has been put toward analyzing and mitigating gender bias in word embeddings (Bolukbasi et al., 2016; Zhao et al., 2018b; Dev and Phillips, 2019; Ethayarajh et al., 2019).”
Automatically Neutralizing Subjective Bias in Text (2019, background): “…prior research which has debiased representations of text by removing dimensions of prejudice from word embeddings (Bolukbasi et al. 2016; Gonen and Goldberg 2019; Ethayarajh, Duvenaud, and Hirst 2019) and the hidden states of predictive models (Zhao et al. 2018; Das, Dantcheva, and Bremond 2018).”
Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases (2020, background): “[22] demonstrate how these limitations of SWE impact measuring gender biases.”
RedditBias: A Real-World Resource for Bias Evaluation and Debiasing of Conversational Language Models (2021, background): “, 2019), analyses of the proposed measures (e.g., Gonen and Goldberg, 2019; Ethayarajh et al., 2019), more comprehensive evaluation frameworks (Lauscher et al.”
Neutralizing Gender Bias in Word Embedding with Latent Disentanglement and Counterfactual Generation (2020, background): “wnstream tasks. 1 Introduction Recent researches have disclosed that the word embeddings contain unexpected biases in their geometry on the embedding space (Bolukbasi et al., 2016; Garg et al., 2018; Ethayarajh et al., 2019; Zhao et al., 2019; Agarwal et al., 2019). The biases reﬂect unwanted stereo”
Can Existing Methods Debias Languages Other than English? First Attempt to Analyze and Mitigate Japanese Word Embeddings (2020, background): “Ethayarajh et al. (2019) comment on the results of Gonen and Goldberg (2019) stating that Hard Debias removes only the components of the predeﬁned gender direction, and that it is impossible to remove other undeﬁned components of the gender direction.”
Undesirable biases in NLP: Averting a crisis of measurement (2022, background): “…whether operationalizations are consistent with the underlying theory (Blodgett et al., 2021) — or conversely “do not make sense” — or testing whether slight variations to the operationalizations that should not matter, lead to different outcomes (Ethayarajh et al., 2019; Zhang et al., 2020).”
Is Your Classifier Actually Biased? Measuring Fairness under Uncertainty with Bernstein Bounds (2020, background): “Moreover, recent work has found that WEAT (Caliskan et al., 2017), the most popular test of embedding bias, can be easily manipulated to claim that bias is present or absent (Ethayarajh et al., 2019a,b).”
Probabilistic Bias Mitigation in Word Embeddings (2019, methodology): “However, WEAT has been recently shown to overestimate bias as a result of implicitly relying on similar frequencies for the target words [5], and Gonen and Goldberg [6] demonstrated that evidence of bias can still be recovered after geometric bias mitigation by examining the neighborhood of a target”
On the Relationships Between the Grammatical Genders of Inanimate Nouns and Their Co-Occurring Adjectives and Verbs (2020, background): “Finally, many scholars have proposed a variety of computational techniques for mitigating gender norms and stereotypes in a wide range of languagebased applications (Dev and Phillips, 2019; Dinan et al., 2019; Ethayarajh et al., 2019; Hall Maudslay et al., 2019; Stanovsky et al., 2019; Tan and Celis”
Evaluating Large Language Models (2022, background): “Numerous works in the community demonstrate why existing bias measurements are flawed [3] or brittle [11].”
Evaluating Gender Bias in Hindi-English Machine Translation (2021, background): “While these metrics can be used to quantify gender bias, despite not necessarily being robust, as is illustrated in Ethayarajh et al. (2019) which delves into the ﬂaws of WEAT, they also treat gender in binary terms, which is also a consistent trend across research related to the ﬁeld.”
AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddings (2020, methodology): “, 2002), has recently been shown to systematically overestimate the bias present in an embedding space (Ethayarajh et al., 2019), in this work, we couple it with several other bias tests, designed to capture and quantify other aspects of human biases: Embedding Coherence Test (Dev and Phillips, 2019”
The Birth of Bias: A case study on the evolution of gender bias in an English language model (2022, methodology): “Ethayarajh et al. (2019) and Du et al. (2021) warn that including low-frequency words can negatively impact the bias measure, which we suspect is the case here.”
Utility Is in the Eye of the User: A Critique of NLP Leaderboard Design (2020, background): “Many types of bias have also been found in NLP systems, with models performing better on genderstereotypical inputs (Rudinger et al., 2018; Ethayarajh, 2020) and racial stereotypes being captured in embedding space (Manzini et al., 2019; Ethayarajh et al., 2019a,b; Ethayarajh, 2019).”