The Tail Wagging the Dog: Dataset Construction Biases of Social Bias Benchmarks

By Nikil Selvam, Sunipa Dev, Daniel Khashabi, Tushar Khot, Kai-Wei Chang. 2022

34 citations recorded by the source corpus.

How reliably can we trust the scores obtained from social bias benchmarks as faithful indicators of problematic social biases in a given model? In this work, we study this question by contrasting social biases with non-social biases that stem from choices made during dataset construction (which might not even be discernible to the human eye). To do so, we empirically simulate various alternative constructions for a given benchmark based on seemingly innocuous modifications (such as paraphrasing or random-sampling) that maintain the essence of their social bias. On two well-known social bias benchmarks (Winogender and BiasNLI), we observe that these shallow modifications have a surprising effect on the resulting degree of bias across various models and consequently the relative ordering of these models when ranked by measured bias. We hope these troubling observations motivate more robust measures of social biases.

Subfields: Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

1. Barret Zoph
2. Denny Zhou
3. Yi Tay
4. Jason Wei
5. Xinyun Chen
6. Adam Roberts
7. Slav Petrov
8. Albert Webson

In Safety, Robustness & Interpretability

1. Shixiang Shane Gu
2. Quoc V. Le
3. Jason Wei
4. Xinyun Chen
5. Yi Tay
6. Adam Roberts
7. Ed H. Chi
8. Barret Zoph

Citations in the corpus

• Scaling Instruction-Finetuned Language Models (2022, background): “In this work we look at only one formulation of the task framing, which may not generalize to other prompts or usage contexts (Selvam et al., 2022).”
• Bias and Fairness in Large Language Models: A Survey (2023, background): “As we discuss in Section 4, several widely-used evaluation datasets suffer from reliability and validity issues, including ambiguities about whether instances accurately reflect real-world stereotypes, inconsistent treatment of social groups, assumptions of near-perfect understanding of language, an”