It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks

By Michelle Bao, Angela Zhou, Samantha Zottola, Brian Brubach, Sarah L. Desmarais, Aaron Horowitz, Kristian Lum, Suresh Venkatasubramanian. 2021

118 citations recorded by the source corpus.

Risk assessment instrument (RAI) datasets, particularly ProPublica's COMPAS dataset, are commonly used in algorithmic fairness papers due to benchmarking practices of comparing algorithms on datasets used in prior work. In many cases, this data is used as a benchmark to demonstrate good performance without accounting for the complexities of criminal justice (CJ) processes. However, we show that pretrial RAI datasets can contain numerous measurement biases and errors, and due to disparities in discretion and deployment, algorithmic fairness applied to RAI datasets is limited in making claims about real-world outcomes. These reasons make the datasets a poor fit for benchmarking under assumptions of ground truth and real-world impact. Furthermore, conventional practices of simply replicating previous data experiments may implicitly inherit or edify normative positions without explicitly interrogating value-laden assumptions. Without context of how interdisciplinary fields have engaged in CJ research and context of how RAIs operate upstream and downstream, algorithmic fairness practices are misaligned for meaningful contribution in the context of CJ, and would benefit from transparent engagement with normative considerations and values related to fairness, justice, and equality. These factors prompt questions about whether benchmarks for intrinsically socio-technical systems like the CJ system can exist in a beneficial and ethical way.

Subfields: Artificial Intelligence, Safety Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Safety, Robustness & Interpretability

Citations in the corpus

Predictability and Surprise in Large Generative Models (2022, methodology): “4; however, we foreground here that benchmark risk assessment instrument datasets like COMPAS often contain numerous measurement biases and errors which can make them ill-suited for making claims about real-world impact without carefully considering the the complicated socio-technical systems (in th”
Algorithm fairness in artificial intelligence for medicine and healthcare (2023, background): “These factors can also vary by location: the same person may be considered of different races in different geographic locations, as exemplified by self-reported Asian ethnicity in the TCGA and PIONEER cohorts, and by self-reported race in the COMPAS algorithm 146,201 .”
REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets (2020, unknown): “Beyond vision datasets, biases in datasets from other domains have also been interrogated, such as those from criminal justice (Bao et al. 2021), the U.”
Retiring Adult: New Datasets for Fair Machine Learning (2021, background): “The distinction we draw between benchmark data and substantive domain-specific investigations resonates with recent work that points out issues with using data about risk assessments tools from the criminal justice domain as machine learning benchmarks [4].”