Is Your Classifier Actually Biased? Measuring Fairness under Uncertainty with Bernstein Bounds

By Kawin Ethayarajh. 2020

27 citations recorded by the source corpus.

Most NLP datasets are not annotated with protected attributes such as gender, making it difficult to measure classification bias using standard measures of fairness (e.g., equal opportunity). However, manually annotating a large dataset with a protected attribute is slow and expensive. Instead of annotating all the examples, can we annotate a subset of them and use that sample to estimate the bias? While it is possible to do so, the smaller this annotated sample is, the less certain we are that the estimate is close to the true bias. In this work, we propose using Bernstein bounds to represent this uncertainty about the bias estimate as a confidence interval. We provide empirical evidence that a 95% confidence interval derived this way consistently bounds the true bias. In quantifying this uncertainty, our method, which we call Bernstein-bounded unfairness, helps prevent classifiers from being deemed biased or unbiased when there is insufficient evidence to make either claim. Our findings suggest that the datasets currently used to measure specific biases are too small to conclusively identify bias except in the most egregious cases. For example, consider a co-reference resolution system that is 5% more accurate on gender-stereotypical sentences – to claim it is biased with 95% confidence, we need a bias-specific dataset that is 3.8 times larger than WinoBias, the largest available.

Subfields: Artificial Intelligence, Safety Research, Sociology and Political Science

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Sociology and Political Science

Citations in the corpus

Language (Technology) is Power: A Critical Survey of “Bias” in NLP (2020, neutral): “…et al. (2019); Nozza et al. (2019); Sap et al. (2019); Sheng et al. (2019); Stanovsky et al. (2019); Vaidya et al. (2019); Webster et al. (2019); Ethayarajh (2020); Gaut et al. (2020); Gencoglu (2020); Hovy et al. (2020); Huang et al. (2020); Kim et al. (2020); Peng et al. (2020); Ravfogel…”
Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities (2021, background): “The resulting biased estimates of queer fairness may involve high levels of uncertainty (Ethayarajh 2020), though it may be possible to utilise unlabeled data for tightening the bounds (Ji, Smyth, and Steyvers 2020).”
Undesirable biases in NLP: Averting a crisis of measurement (2022, unknown): “While the replicability of results is also a potential concern for NLP bias measures, we refer interested readers to other work (e.g., Ethayarajh, 2020), and instead focus our discussion on reliability in the psychometric sense.”
Utility Is in the Eye of the User: A Critique of NLP Leaderboard Design (2020, background): “Many types of bias have also been found in NLP systems, with models performing better on genderstereotypical inputs (Rudinger et al., 2018; Ethayarajh, 2020) and racial stereotypes being captured in embedding space (Manzini et al.”