Fair Prediction with Disparate Impact: A Study of Bias in Recidivism Prediction Instruments

By Alexandra Chouldechova. 2016

2,484 citations recorded by the source corpus.

Abstract Recidivism prediction instruments (RPIs) provide decision-makers with an assessment of the likelihood that a criminal defendant will reoffend at a future point in time. Although such instruments are gaining increasing popularity across the country, their use is attracting tremendous controversy. Much of the controversy concerns potential discriminatory bias in the risk assessments that are produced. This article discusses several fairness criteria that have recently been applied to assess the fairness of RPIs. We demonstrate that the criteria cannot all be simultaneously satisfied when recidivism prevalence differs across groups. We then show how disparate impact can arise when an RPI fails to satisfy the criterion of error rate balance.

Subfields: Nature and Landscape Conservation, Sociology and Political Science, Strategy and Management, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Nature and Landscape Conservation

In Sociology and Political Science

In Strategy and Management

In Safety, Robustness & Interpretability

Citations in the corpus

Model Cards for Model Reporting (2018, background): “or different errors that can be calculated from the confusion matrix for binary classification systems. • These also correspond to metrics in recent definitions of “fairness” in machine learning (cf. [8, 25]), where parity across subgroups for different metrics correspond to different fairness crite”
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs (2021, background): “searchers have drawn on ProPublica’s design choices, evaluation dataset, and source code to further interrogate its results and conclusions (e.g., Kleinberg, Mullainathan, and Raghavan 2017a; Corbett-Davies et al. 2016; Chouldechova 2017).”
Measurement and Fairness (2019, background): “r operationalization, i.e., calibration. The fairness in machine learning community responded, largely re ecting the incompatibilities and shortcomings of dierent fairness operationalizations (e.g., [8, 11, 15, 13, 39, 54]). Meanwhile, Chouldechova [11] and Kleinberg et al. [39] proved that the two ”
Artificial Intelligence, Values, and Alignment (2020, background): “In the United States, for example, parole-recommendation algorithms have been criticised for departing from notions of fairness that are foundational to the criminal justice system (Chouldechova, 2017; Koepke and Robinson, 2018).”
Does Object Recognition Work for Everyone? (2019, background): “Moreover, several impossibility results exist: for instance, equalized odds is incompatible with other formulations of fairness [7, 10, 27], and it is impossible to achieve equalized odds using a calibrated classifier without withholding a randomly selected subset of the classifier’s predictions [35”
Examining Gender and Race Bias in Two Hundred Sentiment Analysis Systems (2018, background): “Examples include, loan eligibility and crime recidivism prediction systems that negatively assess people belonging to a certain pin/zip code (which may disproportionately impact people of a certain race) (Chouldechova, 2017) and resumé sorting systems that believe that men are more qualified to be p”
Inherent Trade-Offs in the Fair Determination of Risk Scores (2016, neutral): “Also concurrent with our work and that of Hardt et al., Chouldechova [5] and Corbett-Davies et al. [6] (and see also [7]) consider binary prediction subject to these same analogues of the balance conditions for the negative and positive classes, together with a form of calibration adapted to binary ”
Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI (2019, positive): “In a certain literature strand, an explainable ML model suggests a clear visualization of the relations aﬀecting a result, allowing for a fairness or ethical analysis of the model at hand [3,100] .”
A Survey on Bias and Fairness in Machine Learning (2019, background): “That is, if for all values of s, P(Y =1|S=s,R=b)=P(Y =1|S=s,R=w) [31].”
Counterfactual Fairness (2017, background): “These criteria can be incompatible in general, as discussed in [1, 7, 22].”
Fair decision making using privacy-protected data (2019, background): “minority language speakers are treated equally) we are also concerned with equitable treatment of arbitrary pairs of jurisdictions. The metric we use for this problem is related to error rate balance [7] but other metrics could also be considered. StatisticalAgencyPractices:Statisticalagenciesliketh”
Learning Adversarially Fair and Transferable Representations (2018, background): “Formally, this is Satisfying these constraints is known to conﬂict with learning well-calibrated classiﬁers (Chouldechova, 2017; Klein-berg et al., 2016; Pleiss et al., 2017).”
The mythos of model interpretability (2016, background): “A popular approach for deep neural nets, saliency maps compute gradients of predictions with respect to inputs, rendering the gradient as an image (Simonyan et al., 2013).”
Fair-by-design explainable models for prediction of recidivism (2019, background): “Such instruments may be used in pre-trial decision-making, prediction of locations where crimes most occur, proﬁles that are more likely to commit violent crimes, as well as to predict people who are likely to reoffend [1, 2].”
Understanding artificial intelligence ethics and safety (2019, background): “08139 Chouldechova, A. (2016). Fair prediction with disparate impact: A study of bias in recidivism prediction instruments. ArXiv:1610.07524. Retrieved from http://arxiv.org/abs/1610.07524 Corbett-Davies, S., Pierson, E., Feller, A., Goel, S., & Huq, A. (2017). Algorithmic decision making and the co”
A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual &Group Unfairness via Inequality Indices (2018, background): “First, several studies have observed that satisfying multiple fairness conditions at the same time is infeasible [7, 9, 22, 23].”