Understanding the Origins of Bias in Word Embeddings

By Marc-Etienne Brunet, Colleen Alkalay-Houlihan, Ashton Anderson, Richard S. Zemel. 2018

226 citations recorded by the source corpus.

The power of machine learning systems not only promises great technical progress, but risks societal harm. As a recent example, researchers have shown that popular word embedding algorithms exhibit stereotypical biases, such as gender bias. The widespread use of these algorithms in machine learning systems, from automated translation services to curriculum vitae scanners, can amplify stereotypes in important contexts. Although methods have been developed to measure these biases and alter word embeddings to mitigate their biased representations, there is a lack of understanding in how word embedding bias depends on the training data. In this work, we develop a technique for understanding the origins of bias in word embeddings. Given a word embedding trained on a corpus, our method identifies how perturbing the corpus will affect the bias of the resulting embedding. This can be used to trace the origins of word embedding bias back to the original training documents. Using our method, one can investigate trends in the bias of the underlying corpus and identify subsets of documents whose removal would most reduce bias. We demonstrate our techniques on both a New York Times and Wikipedia corpus and find that our influence function-based approximations are very accurate.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “…Sutton et al. (2018); Zhao et al. (2018a,b); Agarwal et al. (2019); Basta et al. (2019); Bhaskaran and Bhallamudi (2019); Bordia and Bowman (2019); Brunet et al. (2019); Cao and Daumé (2019); Chaloner and Maldonado (2019); Cho et al. (2019); Dev and Phillips (2019); Font and Costa-jussà (2019);…”
Queens Are Powerful Too: Mitigating Gender Bias in Dialogue Generation (2019, background): “Recent work in natural language processing has found similar biases, such as in word embeddings (Bolukbasi et al., 2016; Brunet et al., 2018; Zhao et al., 2019), object classification (Zhao et al.”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “For example, while not explored in this case study, we think that the finegrained information that Pythia provides on the data seen during training could benefit the promising literature on influence functions to estimate the role of specific training samples on the encoded bias (Brunet et al., 2019”
A Survey on Bias and Fairness in Machine Learning (2019, methodology): “t these debiasing techniques and states that many recent works on debiasing word embeddings have been superficial, that those techniques just hide the bias and don’t actually remove it. A recent work [22] took a new direction and proposed a preprocessing method for the discovery of the problematic d”
A Survey on Bias and Fairness in Natural Language Processing (2022, neutral): “Complementary to augmentation, Brunet et al. (2019) propose removing those documents which contain the most amount of bias.”
Man is to Person as Woman is to Location: Measuring Gender Bias in Named Entity Recognition (2019, background): “Glove) that aims to preserve gender information in some directions of word vectors, while setting other dimensions free from gender inﬂuence (Zhao et al. 2018b) or other data augmentation techniques (Brunet et al. 2019; Zhao et al. 2019). Other work tried to show and address bias in co-reference res”
Do NLP Models Know Numbers? Probing Numeracy in Embeddings (2019, background): “Understanding this, perhaps by tracing numeracy back to the training data, is a fruitful direction to explore further (c.f., inﬂuence functions (Koh and Liang, 2017; Brunet et al., 2019)).”
Algorithmic Fairness (2020, background): “However, previous studies have shown that there are inherent biases in word embeddings (for example, [18, 24, 29, 150]).”
Bias in machine learning - what is it good for? (2020, methodology): “Methods that reduce this kind of bias in word embeddings have been suggested, and either modify already trained word embeddings [5] or remove parts of the data used to train the embeddings [6].”
Influence Functions in Deep Learning Are Fragile (2020, methodology): “In NLP, inﬂuence functions have been used to detect biases in word-embeddings [12] whereas in the domain of ML security, inﬂuence functions have been shown to be effective in crafting stronger data-poisoning attacks [13].”
Fairness in representation: quantifying stereotyping as a representational harm (2019, background): “Some of the research in this area has taken a representational, preprocessing approach to reversing training data skew [17, 26, 40, 6, 16] by changing the inputs so that a classiﬁer ﬁnds fair outcomes.”
Mitigating Gender Bias in Captioning Systems (2020, background): “Image Baseline GAIC GAIC es Figure 3: Qualitative comparison of baseline and our proposed models. 6 Related Work Gender Bias in Dataset: Gender bias in dataset has been studied in a variety of domains[25, 24, 26, 27, 28, 29], especially in natural language processing [10, 30, 31, 32, 33, 34, 11]. Fo”
Trustworthy Artificial Intelligence: A Review (2022, methodology): “In-the-loop 2 Data Bias Word Embedding [27]”
Machine Unlearning of Features and Labels (2021, methodology): “For example, they have been applied to trace bias in word embeddings to documents [24, 25], determine reliable regions in learning models [26], and explain deep neural networks [27].”
Hindi-English Hate Speech Detection: Author Profiling, Debiasing, and Practical Perspectives (2020, background): “Bias Mitigation in Linguistic Deep Learning Models (Brunet et al. 2018) demonstrated the fact that word embed-dings have been shown to contain bias along multiple spec-trums like gender, religion, race, ethnicity that is inherited from their training corpora.”
Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases (2020, background): “Do these unsupervised computer vision models embed human biases like their counterparts in natural language? If so, what are the origins of this bias? In NLP, embedding biases have been traced to word co-occurrences and other statistical patterns in text corpora used for training [12, 10, 7].”
Towards a Comprehensive Understanding and Accurate Evaluation of Societal Biases in Pre-Trained Transformers (2021, background): “After the seminal work of Bolukbasi et al. (2016), bias has been found ubiquitous in word embedding models (Amorim et al., 2018; Brunet et al., 2018; Rudinger et al., 2018; Zhao et al., 2017; Costa-juss`a et al., 2019; Silva et al., 2020).”
Considerations for the Interpretation of Bias Measures of Word Embeddings (2019, background): “The genesis of these biases via the underlying text corpus is explored in depth in (Brunet et al. 2018).”