Measuring Model Biases in the Absence of Ground Truth

By Osman Aka, Ken Burke, Alex Bauerle, Christina Greer, Margaret Mitchell. 2021

42 citations recorded by the source corpus.

The measurement of bias in machine learning often focuses on model performance across identity subgroups (such as man and woman) with respect to groundtruth labels. However, these methods do not directly measure the associations that a model may have learned, for example between labels and identity subgroups. Further, measuring a model's bias requires a fully annotated evaluation dataset which may not be easily available in practice. We present an elegant mathematical solution that tackles both issues simultaneously, using image classification as a working example. By treating a classification model's predictions for a given image as a set of labels analogous to a "bag of words", we rank the biases that a model has learned with respect to different identity labels. We use man, woman as a concrete example of an identity label set (although this set need not be binary), and present rankings for the labels that are most biased towards one identity or the other. We demonstrate how the statistical properties of different association metrics can lead to different rankings of the most "gender biased" labels, and conclude that normalized pointwise mutual information (nPMI) is most useful in practice. Finally, we announce an open-sourced nPMI visualization tool using TensorBoard.

Subfields: Evaluation & Benchmarks, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

In AI Ethics, Fairness & Society

Citations in the corpus

Re-contextualizing Fairness in NLP: The Case of India (2022, methodology): “We obtain similar trends for nPMI (Aka et al., 2021) metric, and a window size of 2, i.”
Scaling Vision Transformers (2021, methodology): “We have conducted sensitive category association analysis as described in [1].”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “For each study, we sweep over guidance values of [1, 3, 5, 7, 8, 10, 12, 15, 18]”
Measuring Data (2022, methodology): “…et al., 2018; Bommasani and Cardie, 2020a); “statistical tendencies”, described as “present” in datasets (Meister and Cotterell, 2021a); “statistical properties” to surface issues with dataset contents (Paullada et al., 2021), and “bias measurements” to quantify dataset bias (Aka et al., 2021).”
Stable Bias: Evaluating Societal Representations in Diffusion Models (2023, unknown): “While it has been well-established that text models and image models encode problematic biases independently (see [13, 12, 15, 2], among many others), the rapid increase in multimodal models (e.g., text-to-image, image-to-text, text-to-audio, etc.) has outpaced the development of approaches for…”
BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling (2022, methodology): “We performed a basic ad-hoc bias analysis looking into possible shortcomings of our models (Nissim, van Noord, and van der Goot, 2020; Blodgett et al., 2020; Aka et al., 2021; Bender et al., 2021).”
Scaling Vision Transformers to 22 Billion Parameters (2023, methodology): “In order to train ViT-22B, we conducted an analysis of sensitive category associations on the JFT-4B dataset as described in Aka et al. (2021).”
Handling Bias in Toxic Speech Detection: A Survey (2022, background): “that use human-annotated dialect labels or do not require ground truth for measuring bias [1]”