CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models

By Nikita Nangia, Clara Vania, Rasika Bhalerao, Samuel R. Bowman. 2020

993 citations recorded by the source corpus.

Pretrained language models, especially masked language models (MLMs) have seen success across many NLP tasks. However, there is ample evidence that they use the cultural biases that are undoubtedly present in the corpora they are trained on, implicitly creating harm with biased representations. To measure some forms of social bias in language models against protected demographic groups in the US, we introduce the Crowdsourced Stereotype Pairs benchmark (CrowS-Pairs). CrowS-Pairs has 1508 examples that cover stereotypes dealing with nine types of bias, like race, religion, and age. In CrowS-Pairs a model is presented with two sentences: one that is more stereotyping and another that is less stereotyping. The data focuses on stereotypes about historically disadvantaged groups and contrasts them with advantaged groups. We find that all three of the widely-used MLMs we evaluate substantially favor sentences that express stereotypes in every category in CrowS-Pairs. As work on building less biased models advances, this dataset can be used as a benchmark to evaluate progress.

Subfields: Artificial Intelligence, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “This was also observed in previous work (Zhang et al., 2022 Table 12: CrowS-Pairs.”
Training language models to follow instructions with human feedback (2022, background): “InstructGPT does not significantly improve over GPT-3 on the Winogender (Rudinger et al., 2018) and CrowSPairs (Nangia et al., 2020) datasets.”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “On the CrowS-Pairs MoE models perform slightly better (less biased) than dense models on average but the difference is not statistically significant (see Table 5 and Table 6).”
Ethical and social risks of harm from Language Models (2021, methodology): “The CrowS-Pairs benchmark finds that cultural stereotypes were reproduced by likelihood estimates of masked LMs BERT (Devlin et al., 2019), and RoBERTA (Liu et al., 2019; Nangia et al., 2020)3.”
Stereotyping Norwegian Salmon: An Inventory of Pitfalls in Fairness Benchmark Datasets (2021, background): “, 2020) and CrowS-Pairs (Nangia et al., 2020))—and coreference resolution— where paired contrastive sentences differ by a gendered pronoun (Winogender (Rudinger et al.”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “Finally, we evaluate on Crows Pairs (Nangia et al., 2020) formulated as a boolean QA task about whether a sentence illustrates a stereotype or not (using a single prompt), and see a deterioration in performance on OPT-IML 175B over OPT, but not on the 30B model.”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “As a preliminary study into the biases learned by BLOOM, we present evaluation on the multilingual CrowS-Pairs dataset, which combines a revised version of the CrowS-Pairs dataset developed by Nangia et al. (2020) together with the French version of CrowS-Pairs introduced by Névéol et al. (2022).”
OPT: Open Pre-trained Transformer Language Models (2022, background): “Given this is a primary data source for OPT-175B, the model may have learned more discriminatory associations, which directly impacts its performance on CrowS-Pairs.”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Popular evaluations for biases use association tests [46] or examine stereotypes [157, 156, 138], correlations and co-occurrences [272], and sentiment analysis [66].”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, methodology): “Nangia et al. (2020) also focus on stereotypes dealing with 9 types of social bias, including race, gender, re-ligion and other factors.”
The Capacity for Moral Self-Correction in Large Language Models (2023, background): “There are many benchmarks for measuring stereotype bias that we do not use in our work [32, 37, 38, 65], along with cogent criticism [9, 10] of these benchmarks and the ones we do use.”
An Empirical Study of Metrics to Measure Representational Harms in Pre-Trained Language Models (2023, background): “Previous work have leveraged power dynamics between two groups to quantify representational harms (Zhao et al., 2018; Rudinger et al., 2018; Zhao et al., 2019; Vig et al., 2020; Nadeem et al., 2021; Nangia et al., 2020).”
Re-contextualizing Fairness in NLP: The Case of India (2022, methodology): “, 2020) and CrowS-P (Nangia et al., 2020) to evaluate models, but they may not capture the stereotypes relevant to India.”
Fewer Errors, but More Stereotypes? The Effect of Model Size on Gender Bias (2022, methodology): “In this section we study this question in a controlled manner, and observe a somewhat surprising trend: depending on the setup for measuring gender bias, conflicting results are observed; on the one hand, in MLM setup larger models are more sensitive to gender bias than smaller models.”
“I’m sorry to hear that”: Finding New Biases in Language Models with a Holistic Descriptor Dataset (2022, methodology): “Our RoBERTa pseudo-log-likelihood calculation adapts the code of Nangia et al. (2020).”
Hi, my name is Martha: Using names to measure and mitigate bias in generative dialogue models (2021, methodology): “Table 9 shows the performance of our de-biased models on pairs of more- and less-stereotypical sentences from StereoSet (Nadeem et al., 2020) and CrowS-Pairs (Nangia et al., 2020) along the axis of gender bias.10 The bias in the original BlenderBot3B model is mitigated for all lower- 10We use these ”
BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generation (2021, methodology): “[26] presented a dataset, similar in spirit to the StereoSet, with 1,508 sentence pairs in which one sentence is more stereotypical than other.”