StereoSet: Measuring stereotypical bias in pretrained language models

By Moin Nadeem, Anna Bethke, Siva Reddy. 2020

1,424 citations recorded by the source corpus.

A stereotype is an over-generalized belief about a particular group of people, e.g., Asians are good at math or African Americans are athletic. Such beliefs (biases) are known to hurt target groups. Since pretrained language models are trained on large real-world data, they are known to capture stereotypical biases. It is important to quantify to what extent these biases are present in them. Although this is a rapidly growing area of research, existing literature lacks in two important aspects: 1) they mainly evaluate bias of pretrained language models on a small set of artificial sentences, even though these models are trained on natural data 2) current evaluations focus on measuring bias without considering the language modeling ability of a model, which could lead to misleading trust on a model even if it is a poor language model. We address both these problems. We present StereoSet, a large-scale natural English dataset to measure stereotypical biases in four domains: gender, profession, race, and religion. We contrast both stereotypical bias and language modeling ability of popular models like BERT, GPT-2, RoBERTa, and XLnet. We show that these models exhibit strong stereotypical biases. Our data and code are available at https://stereoset.mit.edu.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, neutral): “…and Ungar (2019); Stanovsky et al. (2019); Tan and Celis (2019); Zhao et al. (2019); Zhou et al. (2019); Chopra et al. (2020); Gyamfi et al. (2020); Nadeem et al. (2020); Nissim et al. (2020); Papakyriakopoulos et al. (2020); Popović et al. (2020); Ravfogel et al. (2020); Ross et al. (2020);…”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “On the StereoSet benchmark, corresponding dense and sparse models (comparable FLOPs) yield comparable performance.”
Ethical and social risks of harm from Language Models (2021, background): “, 2019) exhibit ‘strong stereotypical associations’ (Nadeem et al., 2020).”
Persistent Anti-Muslim Bias in Large Language Models (2021, background): “As a result, even though the various language models have different architectures, since they are trained on similar corpora of text, they often learn similar biases [9].”
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets (2021, methodology): “Consider the illustrative example about “Norwe-gian salmon” in Figure 1—drawn from an existing benchmark (Nadeem et al., 2020)—which depicts a test meant (according to the metadata) to capture a stereotype about race.”
OPT: Open Pre-trained Transformer Language Models (2022, methodology): “To account for a potential trade-off between bias detection and language modeling capability, StereoSet includes two metrics: Results are shown in Table 5.”
CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models (2020, methodology): “Concurrent to this work, Nadeem et al. (2020) introduce StereoSet, a crowdsourced dataset for associative contexts aimed to measure 4 types of social bias—race, gender, religion, and profession— in language models, both at the intrasentence level, and at the intersentence discourse level.”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Popular evaluations for biases use association tests [46] or examine stereotypes [157, 156, 138], correlations and co-occurrences [272], and sentiment analysis [66].”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, methodology): “We examine the main meth-ods employed in LMs safety evaluation, including model preference-based safety testing (Nadeem et al., 2021; Xu et al., 2023a), adversarial safety attacks (Perez et al., 2022; Ganguli et al., 2022), safety risk detection (Dinan et al., 2019; Sun et al., 2022), and other…”
The Capacity for Moral Self-Correction in Large Language Models (2023, background): “There are many benchmarks for measuring stereotype bias that we do not use in our work [32, 37, 38, 65], along with cogent criticism [9, 10] of these benchmarks and the ones we do use.”
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, methodology): “…the manifestation of societal biases in language models and proposed metrics and datasets to quantify them based on sentiment (Kurita et al., 2019), regard (Sheng et al., 2019), stereotypes (Zhao et al., 2019; Nadeem et al., 2021), style (Smith et al., 2022), or morality (Schramowski et al., 2022).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, positive): “Evidence for this behavior has been substantially corroborated in NLP literature (Nadeem et al., 2021; Nozza et al., 2021; Huang et al., 2020b; Hassan et al., 2021; Blodgett et al., 2020; Liang et al., 2021; Field et al., 2021, inter alia).”
Re-contextualizing Fairness in NLP: The Case of India (2022, methodology): “Religion: Religious biases have been studied in NLP (Dev et al., 2020; Nadeem et al., 2020; Abid et al., 2021), however the social disparities and stereotypes about various religious groups differ significantly in India from the West, (Malik et al., 2021).”
Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias (2022, methodology): “In this section we study this question in a controlled manner, and observe a somewhat surprising trend: depending on the setup for measuring gender bias, conflicting results are observed; on the one hand, in MLM setup larger models are more sensitive to gender bias than smaller models.”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, methodology): “Such prompts are generally derived either from crowdsourcing (Nadeem et al., 2021; Nangia et al., 2021) or from slotting a set of terms into templates (Kurita et al.”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, methodology): “Table 9 shows the performance of our de-biased models on pairs of more- and less-stereotypical sentences from StereoSet (Nadeem et al., 2020) and CrowS-Pairs (Nangia et al., 2020) along the axis of gender bias.10 The bias in the original BlenderBot3B model is mitigated for all lower- 10We use these ”
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation (2021, methodology): “[25] presented a dataset, StereoSet, with 17,000 sentences that measure an LM’s preference for texts expressing stereotypes.”