Measuring Bias in Contextualized Word Representations

By Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W. Black, Yulia Tsvetkov. 2019

517 citations recorded by the source corpus.

Contextual word embeddings such as BERT have achieved state of the art performance in numerous NLP tasks. Since they are optimized to capture the statistical properties of training data, they tend to pick up on and amplify social stereotypes present in the data as well. In this study, we (1) propose a template-based method to quantify bias in BERT; (2) show that this method obtains more consistent results in capturing social biases than the traditional cosine based method; and (3) conduct a case study, evaluating gender bias in a downstream task of Gender Pronoun Resolution. Although our case study focuses on gender bias, the proposed technique is generalizable to unveiling other biases, including in multiclass settings, such as racial and religious biases.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, background): “…Font and Costa-jussà (2019); Gonen and Goldberg (2019); James-Sorenson and Alvarez-Melis (2019); Kaneko and Bollegala (2019); Karve et al. (2019); Kurita et al. (2019); Lauscher and Glavaš (2019); Lee et al. (2019); Manzini et al. (2019); Mayfield et al. (2019); Précenth (2019); Pujari et al.…”
StereoSet: Measuring stereotypical bias in pretrained language models (2020, background): “Later, Kurita et al. (2019) show that cosine similarity is not the best association metric and deﬁne a new association metric based on the probability of predicting an attribute given the target in generic sentential context, e.g., [target] is [mask] , where [mask] is the attribute.”
LaMDA: Language Models for Dialog Applications (2022, background): “, [58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72]), and inappropriately revealing personally identifying information (PII) from training data [73].”
Ethical and social risks of harm from Language Models (2021, background): “The StereoSet benchmark measures references to stereotypes of race, gender, religion, and profession in generative LMs and finds that the models GPT2 (Radford et al., 2018b) and masked models BERT (Devlin et al., 2019), ROBERTA (Liu et al., 2019), XLNET (Yang et al., 2019) exhibit ‘strong stereotypi”
RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models (2020, background): “A wealth of work has shown that toxicity and social biases in training data are acquired by large pretrained sentence encoders (e.g., gender bias in BERT; May et al., 2019; Zhao et al., 2019; Basta et al., 2019; Kurita et al., 2019).”
Measuring and Improving BERT’s Mathematical Abilities by Predicting the Order of Reasoning. (2021, background): “Even though such language models have initially shown promising results, more recent investigation shows they may rely on various biases in their predictions (Hendricks et al., 2018; Brown et al., 2020; Bhardwaj et al., 2020; Kurita et al., 2019).”
Social Biases in NLP Models as Barriers for Persons with Disabilities (2020, methodology): “Following prior work (Kurita et al., 2019) studying social biases in BERT, we adopt a templatebased fill-in-the-blank analysis.”
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets (2021, background): “…formats, including sentence templates where terms pertaining to groups or their attributes are perturbed (Rudinger et al., 2018; Zhao et al., 2018; Kurita et al., 2019), prompts designed to elicit problematic responses (Gehman et al., 2020; Sheng et al., 2019; Example Sentences Context I really…”
The Capacity for Moral Self-Correction in Large Language Models (2023, background): “Large language models exhibit harmful social biases [1, 6, 8, 11, 15, 24, 29, 50, 62] that can sometimes get worse for larger models [2, 18, 20, 43, 55].”
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, background): “…have highlighted the manifestation of societal biases in language models and proposed metrics and datasets to quantify them based on sentiment (Kurita et al., 2019), regard (Sheng et al., 2019), stereotypes (Zhao et al., 2019; Nadeem et al., 2021), style (Smith et al., 2022), or morality…”
Galactica: A Large Language Model for Science (2022, background): “Models may mirror misinformation, stereotypes and bias in the corpus (Sheng et al., 2019; Kurita et al., 2019; Dev et al., 2019; Blodgett et al., 2020; Sheng et al., 2021).”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, background): “Large language models exhibit a wide range of harmful behaviors such as reinforcing social biases (e.g., [47, 28, 1, 33, 7]), generating offensive or toxic outputs [25], leaking personally identiﬁable information from the training data [13], aiding in disinformation campaigns [12], generating…”
Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias (2022, methodology): “Kurita et al. (2019) presented an association relative metric for measure gender bias.”
Towards Understanding and Mitigating Social Biases in Language Models (2021, methodology): “Recent work has also extended this analysis towards measuring (Tan and Celis, 2019; Guo and Caliskan, 2020; Kurita et al., 2019) and mitigating (Liang et al.”
Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Models (2021, methodology): “Indeed, since contextual word embedding models such as BERT are optimised to capture the statistical properties of training data, they tend to pick up and amplify any social stereotypes that may be present (Kurita et al., 2019).”
BERTology Meets Biology: Interpreting Attention in Protein Language Models (2020, background): “However, the strong performance of the Transformer comes at the cost of interpretability, and this lack of transparency can hide underlying problems such as model bias and spurious correlations [41, 55, 75].”
Towards Controllable Biases in Language Generation (2020, background): “Much of the work on social biases in natural language processing focuses on biases in word embeddings (Bolukbasi et al., 2016; Zhao et al., 2018b, 2019; Kurita et al., 2019) and natural language understanding tasks (Rudinger et al.”