Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets

By Su Lin Blodgett, Gilsinia Lopez, Alexandra Olteanu, Robert Sim, Hanna M. Wallach. 2021

389 citations recorded by the source corpus.

Auditing NLP systems for computational harms like surfacing stereotypes is an elusive goal. Several recent efforts have focused on benchmark datasets consisting of pairs of contrastive sentences, which are often accompanied by metrics that aggregate an NLP system’s behavior on these pairs into measurements of harms. We examine four such benchmarks constructed for two NLP tasks: language modeling and coreference resolution. We apply a measurement modeling lens—originating from the social sciences—to inventory a range of pitfalls that threaten these benchmarks’ validity as measurement models for stereotyping. We find that these benchmarks frequently lack clear articulations of what is being measured, and we highlight a range of ambiguities and unstated assumptions that affect how these benchmarks conceptualize and operationalize stereotyping.

Subfields: Artificial Intelligence, General Social Sciences, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “…et al., 2021b), there is a lack of standardization of fairness benchmarks, an understanding of what harms different bias measures in NLP relate to (Blodgett et al., 2020, 2021; Jacobs & Wallach, 2021), and coverage of identities in fluid, comprehensive ways (Cao & Daumé III, 2020; Dev et al.,…”
Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “This could imply that while there is less frequent context about She pronouns, comments about “females” are more prevalent, perhaps reflecting the differences in linguistic markedness of these terms (Blodgett et al., 2021).”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, background): “…of instances where a model prefers a stereotypical association over an anti-stereotypical association (SS score close to 50 is better, while a more biased 11The two benchmarks have limitations such as lack of clear articulations of how certain biases are being measured (Blodgett et al., 2021).”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “The CrowS-Pairs version used here differs from the original by addressing some of the issues pointed out by Blodgett et al. (2021) and by constructing 200 additional sentence pairs based on stereotypes collected from French speakers.”
OPT: Open Pre-trained Transformer Language Models (2022, background): “While there may be shortcomings in these benchmarks (Blodgett et al., 2021; Ja-cobs and Wallach, 2021), these measurements provide a ﬁrst step towards understanding the limitations of OPT-175B.”
The Capacity for Moral Self-Correction in Large Language Models (2023, background): “There are many benchmarks for measuring stereotype bias that we do not use in our work [32, 37, 38, 65], along with cogent criticism [9, 10] of these benchmarks and the ones we do use.”
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, methodology): “Recently, Blodgett et al. (2021) exposed several issues in the reliability of both WB and WG datasets.”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, background): “For most NLP tasks, crowdsourced data usually makes up for its size in quality; however, as mentioned above, Blodgett et al. (2021) outlined several data quality issues arising from crowdsourcing socially relevant data.”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, background): “…datasets for their directional estimate of relative levels of bias, despite some limitations (inconsistencies, data quality issues, etc.) as noted by Blodgett et al. (2021). capacity versions of BlenderBot (including those distilled from BlenderBot3B), and bias is likewise suppressed for all…”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “4 While previous works have found the original version of CrowS-Pairs (Nangia et al., 2020) benchmark of questionable validity (Blodgett et al., 2021), N´ev´eol et al. (2022) have revised the English dataset to take care of the raised concerns.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, background): “Notwithstanding their deficiency (Blodgett et al., 2021; Jacobs & Wallach, 2021), these datasets serve as a meaningful initial step towards an open quantitative evaluation LLMs.”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, background): “According to Jacobs and Wallach (2021) and Blodgett et al. (2021), bias mea-452 sures are measurement models which link observable properties, e.g., quality-of-service and representational biases, with unobservable theoretical constructs such as social discrimination, power dynamics, and systemic…”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “However, as noted in Blodgett et al. [17], some stereotypes are harmful and should not be output at all, regardless of the paired anti-stereotype’s relative likelihood.”
A Survey of Race, Racism, and Anti-Racism in NLP (2021, background): “More work is needed to understand any privacy concerns and the strengths and limitations of these data (Blodgett et al., 2021).”
French CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than English (2022, methodology): “During the translation process, we identiﬁed issues with the original CrowS-pairs content, including some described in Blodgett et al. (2021).”
Marked Personas: Using Natural Language Prompts to Measure Stereotypes in Language Models (2023, positive): “In line with previous work, we define stereotypes as traits that have been documented to be broadly associated with a demographic group in ways that reify existing social hierarchies (Deaux and Kite, 1993; Heilman, 2001; Caliskan et al., 2017; Blodgett et al., 2021; Weidinger et al., 2021).”