Assessing Social and Intersectional Biases in Contextualized Word Representations

By Y. Tan, Elisa Celis. 2019

264 citations recorded by the source corpus.

Social bias in machine learning has drawn significant attention, with work ranging from demonstrations of bias in a multitude of applications, curating definitions of fairness for different contexts, to developing algorithms to mitigate bias. In natural language processing, gender bias has been shown to exist in context-free word embeddings. Recently, contextual word representations have outperformed word embeddings in several downstream NLP tasks. These word representations are conditioned on their context within a sentence, and can also be used to encode the entire sentence. In this paper, we analyze the extent to which state-of-the-art models for contextual word representations, such as BERT and GPT-2, encode biases with respect to gender, race, and intersectional identities. Towards this, we propose assessing bias at the contextual word level. This novel approach captures the contextual effects of bias missing in context-free word embeddings, yet avoids confounding effects that underestimate bias at the sentence encoding level. We demonstrate evidence of bias at the corpus level, find varying evidence of bias in embedding association tests, show in particular that racial bias is strongly encoded in contextual word models, and observe that bias effects for intersectional minorities are exacerbated beyond their constituent minority identities. Further, evaluating bias effects at the contextual word level captures biases that are not captured at the sentence level, confirming the need for our novel approach.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, methodology): “…et al. (2019); Précenth (2019); Prost et al. (2019); Pujari et al. (2019); Qian et al. (2019); Sedoc and Ungar (2019); Stanovsky et al. (2019); Tan and Celis (2019); Zhao et al. (2019); Zhou et al. (2019); Chopra et al. (2020); Gyamfi et al. (2020); Nadeem et al. (2020); Nissim et al. (2020);…”
Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies (2021, neutral): “…driven by model and dataset biases due to tainted examples, limited features, and sample size disparities (Wang et al., 2019; Barocas et al., 2019; Tan and Celis, 2019), which are consequences of the non-recognition and a lack of understanding of non-binary genders in society (MAP, 2016; Rajunov…”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, methodology): “account, templates were necessary for measuring social biases, such as stereotypical association with other text content (Tan and Celis, 2019).”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, background): “…for names to refer to individuals with particular demographics—such gender and race (Tzioumis, 2018; Newman et al., 2018) or age (Lieberson, 2000; Twenge et al., 2010)—it is difficult to imagine there being a necessary or causal relationship between your name and most other facts about you,…”
CM3: A Causal Masked Multimodal Model of the Internet (2022, methodology): “…and racial biases that parallel humans through the Word Embedding Association Test (WEAT) (Caliskan et al., 2017), the Sentence Encoder Association Test (SEAT) (May et al., 2019) and the Grounded-WEAT/Grounded-SEAT (Ross et al., 2021) metrics for multimodal language models (Tan & Celis, 2019).”
Alignment of Language Agents (2021, background): “However this doesn’t remove all concerns - for example, there’s also some evidence of bias in WebText, e.g. see Tan and Celis (2019).”
Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models (2021, methodology): “However, the BookCorpus data has since been shown to suffer from similar gender imbalance problems (Tan and Celis, 2019) as has English Wikipedia where, for example, only 15.”
Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias (2020, background): “Gender bias has surfaced as a major concern in word representations, both static word embeddings (Caliskan et al., 2017; Bolukbasi et al., 2016) and contextualized word representations (Zhao et al., 2019a; Basta et al., 2019; Tan and Celis, 2019).”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “However, the strong performance of the Transformer comes at the cost of interpretability, and this lack of transparency can hide underlying problems such as model bias and spurious correlations [39, 53, 73].”
Large image datasets: A pyrrhic win for computer vision? (2020, background): “In [101], the authors analyzed occupation words stereotypically perceived as male (that they termed as M-biased words) as well as occupation words stereotypically perceived as female (F-biased words) in large text corpora and the ensuing downstream effects when used to generate contextual word repre”
UNQOVERing Stereotypical Biases via Underspecified Questions (2020, neutral): “The majority of the work in the area is dedicated to pre-trained models, often via similaritybased analysis of the biases in input representations (Bolukbasi et al., 2016a; Garg et al., 2018; Chaloner and Maldonado, 2019; Bordia and Bowman, 2019; Tan and Celis, 2019; Zhao et al., 2019, 2020), or an ”
Measuring and Reducing Gendered Correlations in Pre-trained Models (2020, neutral): “Recent work has applied association tests to probe for gender bias in contextualized word embeddings (Basta et al., 2019; Peters et al., 2018; Tan and Celis, 2019) with mixed results.”
A Survey of Race, Racism, and Anti-Racism in NLP (2021, background): “A small selection of papers have examined intersectional biases in embeddings or word co-occurrences (Herbelot et al., 2012; May et al., 2019; Tan and Celis, 2019; Lepori, 2020), but we did not identify mentions of intersectionality in any other NLP research areas.”
Cyberbullying Detection With Fairness Constraints (2020, background): “embeddings (e.g. BERT [28], ELMo [29], GPT-2 [30]), bias in sentence embeddings, bias in machine learning algorithms, or bias in human annotators [31], [32], [33], [34], [35], [36], [37], [38], [39], [40], [41], [42]. For instance, a recent study by Nadeem et al. show that stereotypical bias is ampl”
Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases (2020, methodology): “Tan and Celis [61] adopted the method of May et al.”
Boosting Search Engines with Interactive Agents (2021, neutral): “Broader Impact We would like to note that pre-trained language models of the kind used here have been shown to capture societal biases (Tan & Celis, 2019; Webster et al., 2020), which motivates a broad discussion about potential harms and mitigations (Blodgett et al., 2020; Bender et al., 2021).”
Disembodied Machine Learning: On the Illusion of Objectivity in NLP (2021, background): “As sociolects and dialects may not be equally distributed in training data for contextual models (Dunn, 2020), similar issues of which bodies are embodied plague such models (Tan and Celis, 2019) (7)While ‘context’ here is limited to what is in the sentence, language production is situated within a ”