On the Intrinsic and Extrinsic Fairness Evaluation Metrics for Contextualized Language Representations

By Yang Trista Cao, Yada Pruksachatkun, Kai-Wei Chang, Rahul Gupta, Varun Kumar, Jwala Dhamala, Aram Galstyan. 2022

112 citations recorded by the source corpus.

Multiple metrics have been introduced to measure fairness in various natural language processing tasks. These metrics can be roughly categorized into two categories: 1) extrinsic metrics for evaluating fairness in downstream applications and 2) intrinsic metrics for estimating fairness in upstream contextualized language representation models. In this paper, we conduct an extensive correlation study between intrinsic and extrinsic metrics across bias notions using 19 contextualized language models. We find that intrinsic and extrinsic metrics do not necessarily correlate in their original setting, even when correcting for metric misalignments, noise in evaluation datasets, and confounding factors such as experiment configuration for extrinsic metrics.

Subfields: Artificial Intelligence, Safety Research, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, methodology): “A recent work by Cao et al. (2022) examined the correlation among some of the extrinsic and intrinsic metrics in NLG task.”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “…to extrinsic, “downstream”, tasks well (Jin et al., 2021), it is currently unclear whether intrinsic measurement is sufficient, in particular because intrinsic and extrinsic task-based bias metrics don’t always correlate (Delobelle et al., 2021; GoldfarbTarrant et al., 2021; Cao et al., 2022).”
The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks (2022, background): “…et al., 2021; Goldfarb-Tarrant et al., 2023), not correlated in their findings of bias across intrinsic versus extrinsic techniques (GoldfarbTarrant et al., 2021; Cao et al., 2022), and susceptible to adversarial perturbations (Zhang et al., 2021) and seed word selection (Antoniak and Mimno, 2021).”
Prompting GPT-3 To Be Reliable (2022, background): “Biased models cause severe harm when deployed in real-world applications, especially to the minority groups being discriminated against (Cao et al., 2022).”
On Second Thought, Let’s Not Think Step by Step! Bias and Toxicity in Zero-Shot Reasoning (2022, background): “, 2021), leading to contradictory results (Delobelle et al., 2022; Cao et al., 2022; Goldfarb-Tarrant et al., 2021).”
"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letters (2023, background): “Several approaches (Sheng et al., 2020; Gupta et al., 2022; Liu et al., 2021; Cao et al., 2022) studied how to control NLG models for reducing biases.”
Undesirable biases in NLP: Averting a crisis of measurement (2022, methodology): “…NLP bias measures against data from human behavior: e.g., common knowledge of stereotypes, results from psychological tests (Caliskan et al., 2017; Cao et al., 2022a), or statistics of the gender division for occupations in the USA (Caliskan et al., 2017; Zhao et al., 2018; Webster et al.,…”
Perturbation Augmentation for Fairer NLP (2022, background): “, 2021), and often different fairness metrics fail to agree (Delobelle et al., 2021; Goldfarb-Tarrant et al., 2021; Cao et al., 2022).”
From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Models (2023, background): “As extensive research has shown that machine learning models exhibit societal and political biases (Zhao et al., 2018; Blodgett et al., 2020b; Bender et al., 2021; Ghosh et al., 2021; Shaikh et al., 2022; Li et al., 2022; Cao et al., 2022; Goldfarb-Tarrant et al., 2021; Jin et al., 2021), there has ”
Theory-Grounded Measurement of U.S. Social Stereotypes in English Language Models (2022, background): “Though stereotypes from language models may impact the outputs of downstream systems which are built upon these language models, it is not clear how exactly the stereotypes transfer (Cao et al., 2022).”
This Prompt is Measuring : Evaluating Bias Evaluation in Language Models (2023, background): “Other work has found threats to predictive validity where embedding and language model based measures of bias do not correlate with bias in down-stream applications (Goldfarb-Tarrant et al., 2021; Cao et al., 2022).”
Socially Aware Bias Measurements for Hindi Language Representations (2021, background): “of bias which may not be correlated with bias expressed in downstream tasks (Goldfarb-Tarrant et al., 2019; Cao et al., 2022).”