Intrinsic Bias Metrics Do Not Correlate with Application Bias

By Seraphina Goldfarb-Tarrant, R. Marchant, Ricardo Muñoz Sánchez, Mugdha Pandya, Adam Lopez. 2020

214 citations recorded by the source corpus.

Natural Language Processing (NLP) systems learn harmful societal biases that cause them to amplify inequality as they are deployed in more and more situations. To guide efforts at debiasing these systems, the NLP community relies on a variety of metrics that quantify bias in models. Some of these metrics are intrinsic, measuring bias in word embedding spaces, and some are extrinsic, measuring bias in downstream tasks that the word embeddings enable. Do these intrinsic and extrinsic metrics correlate with each other? We compare intrinsic and extrinsic metrics across hundreds of trained models covering different tasks and experimental conditions. Our results show no reliable correlation between these metrics that holds in all scenarios across tasks and languages. We urge researchers working on debiasing to focus on extrinsic measures of bias, and to make using these measures more feasible via creation of new challenge sets and annotated test data. To aid this effort, we release code, a new intrinsic metric, and an annotated test set focused on gender bias in hate speech.

Subfields: Artificial Intelligence, Natural Language Processing, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In AI Ethics, Fairness & Society

Citations in the corpus

Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “Though some work has studied how bias propagates downstream through language technologies [42, 59], an open challenge in designing benchmarks for language model harms is better understanding which metrics reﬂect harms in deployed use cases.”
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning (2022, background): “, 2021), leading to contradictory results (Delobelle et al., 2022; Cao et al., 2022; Goldfarb-Tarrant et al., 2021).”
Undesirable biases in NLP: Averting a crisis of measurement (2022, background): “…model bias, the validity of a measure is not self-evident and, indeed, critical studies of some existing bias measures have shown many validity issues that threaten their usefulness (Gonen & Goldberg, 2019; Ethayarajh et al., 2019; Blodgett et al., 2020, 2021; Goldfarb-Tarrant et al., 2021, i.a.).”
From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models (2023, background): “As extensive research has shown that machine learning models exhibit societal and political biases (Zhao et al., 2018; Blodgett et al., 2020b; Bender et al., 2021; Ghosh et al., 2021; Shaikh et al., 2022; Li et al., 2022; Cao et al., 2022; Goldfarb-Tarrant et al., 2021; Jin et al., 2021), there has ”
Benchmarking Intersectional Biases in NLP (2022, background): “Moreover, the seemingly debi-ased static embeddings often do not alleviate biases in downstream NLP prediction tasks (Goldfarb-Tarrant et al., 2021).”
Biases in Large Language Models: Origins, Inventory, and Discussion (2023, background): “Research on word embeddings [57] has shown that measures of intrinsic bias (in the embedding space) do not correlate reliably with measures of extrinsic bias in tasks such as hate speech detection and coreference resolution.”
Measuring Fairness with Biased Rulers: A Survey on Quantifying Biases in Pretrained Language Models (2021, background): “, 2019), although this view has been questioned (Goldfarb-Tarrant et al., 2020).”
Perturbation Augmentation for Fairer NLP (2022, background): “, 2021), but others find that intrinsic and extrinsic bias measurements do not correlate (Goldfarb-Tarrant et al., 2021; Cao et al., 2022), raising questions about which approach to trust more.”
Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyond (2021, background): “, 2019) but the extent to which biases in pre-trained models propagate to downstream applications remains unclear (Goldfarb-Tarrant et al., 2021).”
Quantifying Social Biases in NLP: A Generalization and Empirical Comparison of Extrinsic Fairness Metrics (2021, background): “However, such intrinsic metrics have been recently shown not to correlate with application bias (Goldfarb-Tarrant et al., 2021).”