What do Bias Measures Measure?

15 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

PaLM: Scaling Language Modeling with Pathways (2022, background): “…al., 2021b), there is a lack of standardization of fairness benchmarks, an understanding of what harms different bias measures in NLP relate to (Blodgett et al., 2020, 2021; Jacobs & Wallach, 2021), and coverage of identities in fluid, comprehensive ways (Cao & Daumé III, 2020; Dev et al., 2021a).”
BBQ: A hand-built bias benchmark for question answering (2021, methodology): “Other studies have focused on a broader range of social biases, covering not only gender and race-based biases, but also age, religion, sexual orientation, and disability status (see Dev et al. 2021 for a survey).”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, background): “In efforts to address the expression of such social biases, a number of bias evaluation benchmarks have been proposed (Dev et al., 2021b; Zhao et al., 2018; Cao and Daumé III, 2020).”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “Markedness is also relevant when comparing language mentioning marginalized groups to language mentioning majority groups, as is often done in distributional bias benchmarks [35].”