Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases

By Wei Guo, Aylin Caliskan. 2020

290 citations recorded by the source corpus.

With the starting point that implicit human biases are reflected in the statistical regularities of language, it is possible to measure biases in English static word embeddings. State-of-the-art neural language models generate dynamic word embeddings dependent on the context in which the word appears. Current methods measure pre-defined social and intersectional biases that occur in contexts defined by sentence templates. Dispensing with templates, we introduce the Contextualized Embedding Association Test (CEAT), that can summarize the magnitude of overall bias in neural language models by incorporating a random-effects model. Experiments on social and intersectional biases show that CEAT finds evidence of all tested biases and provides comprehensive information on the variance of effect magnitudes of the same bias in different contexts. All the models trained on English corpora that we study contain biased representations. GPT-2 contains the smallest magnitude of overall bias followed by GPT, BERT, and then ELMo, negatively correlating with the contextualization levels of the models. Furthermore, we develop two methods, Intersectional Bias Detection (IBD) and Emergent Intersectional Bias Detection (EIBD), to automatically identify the intersectional biases and emergent intersectional biases from static word embeddings in addition to measuring them in contextualized word embeddings. We present the first algorithmic bias detection findings on how intersectional group members are strongly associated with unique emergent biases that do not overlap with the biases of their constituent minority identities. IBD achieves an accuracy of 81.6% and 82.7%, respectively, when detecting the intersectional biases of African American females and Mexican American females, where the random correct identification rates are 14.3% and 13.3%. EIBD reaches an accuracy of 84.7% and 65.3%, respectively, when detecting the emergent intersectional biases unique to African American females and Mexican American females, where the random correct identification rates are 9.2% and 6.1%. Our results indicate that intersectional biases associated with members of multiple minority groups, such as African American females and Mexican American females, have the highest magnitude across all neural language models.

Subfields: Artificial Intelligence, Natural Language Processing, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In AI Ethics, Fairness & Society

Citations in the corpus

An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, background): “Moreover, several other metrics have been developed for measuring societal biases in contextualized word representation (Kurita et al., 2019; May et al., 2019; Guo and Caliskan, 2021) which are extensions of Word Embedding Association Test (WEAT) (Caliskan et al., 2017).”
Towards Understanding and Mitigating Social Biases in Language Models (2021, methodology): “Recent work has also extended this analysis towards measuring (Tan and Celis, 2019; Guo and Caliskan, 2020; Kurita et al., 2019) and mitigating (Liang et al.”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, background): “…works have highlighted the importance of studying intersec-723 tional bias (Bender et al., 2021; Buolamwini and Gebru, 2018; Field et al., 2021; Guo et al., 2019; Crenshaw, 1991) but little research has been conducted around addressing such biases (Magee et al., 2021; Guo and Caliskan, 2021).”
Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Models (2022, background): “, pleasant/unpleasant words or career/familyrelated words) (e.g., Caliskan et al., 2017; Guo and Caliskan, 2021).”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 (2021, background): “Furthermore, we see the effects of intersectionality [34], where BERT, ELMo, GPT and GPT-2 encode more bias against identities marginalized along more than one dimension than would be expected based on just the combination of the bias along each of the axes [54, 132].”
The rise and potential of large language model based agents: a survey (2025, unknown): “Moreover, biases inherent in training data can inﬁltrate neural networks [540, 541].”
On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations (2022, unknown): “We ﬁnd that scores for CEAT and B-S ENT increases to 0 .”
Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluation (2022, background): “They are commonly formulated in terms of the maximum difference (or ratio) of a performance metric either between one group to that of all [53, 77, 121], or between two groups [40, 46].”
Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biases (2020, background): “Guo and Caliskan [34] dispense with sentence templates entirely, pooling across 𝑛 word-level contextual embeddings for the same token extracted from random sentences.”
Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models (2023, background): “The works listed in Table A1 refer to the following papers: Debiasing (Bolukbasi et al., 2016), CrowS-Pairs (Nangia et al., 2020), Stereoset (Nadeem et al., 2021), S. Bias Frames (Sap et al., 2020), CEAT (Guo and Caliskan, 2021), and ABC (Cao et al., 2022).”
The Ghost in the Machine has an American accent: value conflict in GPT-3 (2022, background): “[56] WVS data can be seen to represent Hume’s “Is” of current world values in a manner that takes in a much more diverse representation than the English-language Internet.”