Algorithmic Decision Making and the Cost of Fairness
By Sam Corbett-Davies, Emma Pierson, Avi Feller, Sharad Goel, Aziz Z Huq. 2017
1,410 citations recorded by the source corpus.
Algorithms are now regularly used to decide whether defendants awaiting trial are too dangerous to be released back into the community. In some cases, black defendants are substantially more likely than white defendants to be incorrectly classified as high risk. To mitigate such disparities, several techniques have recently been proposed to achieve algorithmic fairness. Here we reformulate algorithmic fairness as constrained optimization: the objective is to maximize public safety while satisfying formal fairness constraints designed to reduce racial disparities. We show that for several past definitions of fairness, the optimal algorithms that result require detaining defendants above race-specific risk thresholds. We further show that the optimal unconstrained algorithm requires applying a single, uniform threshold to all defendants. The unconstrained algorithm thus maximizes public safety while also satisfying one important understanding of equality: that all individuals are held to the same standard, irrespective of race. Because the optimal constrained and unconstrained algorithms generally differ, there is tension between improving public safety and satisfying prevailing notions of algorithmic fairness. By examining data from Broward County, Florida, we show that this trade-off can be large in practice. We focus on algorithms for pretrial release decisions, but the principles we discuss apply to other domains, and also to human decision makers carrying out structured decision rules.
Subfields: Artificial Intelligence, Law, Sociology and Political Science, AI Ethics, Fairness & Society
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Mohit Bansal
- Yoshua Bengio
- Sergey Levine
- Jianfeng Gao
- Quoc V. Le
- Yejin Choi
- Noah A. Smith
In Law
- Sharad Goel
- Sam Corbett-Davies
- Alexandra Chouldechova
- Silvia Chiappa
- Emma Pierson
- Manuel Gomez-Rodriguez
- Jon M. Kleinberg
- Krishna P. Gummadi
In Sociology and Political Science
- Silvia Chiappa
- Krishna P. Gummadi
- Sharad Goel
- Sam Corbett-Davies
- Simon Caton
- Aaron Roth
- Christian Haas
- Dana Pessach
In AI Ethics, Fairness & Society
- Dana Pessach
- Erez Shmueli
- Silvia Chiappa
- Berk Ustun
- Simon Caton
- Christian Haas
- Adrian Weller
- Aaron Roth
Citations in the corpus
- Measurement and Fairness (2019, background): “This nontrivial measurement task is deeply complex and requires significant debate about justice, policy, and fairness [8, 13, 15, 37].”
- Diagnosing Gender Bias in Image Recognition Systems (2020, methodology): “We formalize this as “conditional demographic parity” (Corbett-Davies et al. 2017).”
- Racial disparities in automated speech recognition (2020, background): “…a problem that has recently come to light in several other advancing applications of machine learning, such as face recognition (5, 6), natural language processing (7–11), online advertising (12, 13), and risk prediction in criminal justice (14–17), healthcare (18, 19), and child services (20, 21).”
- Does Object Recognition Work for Everyone? (2019, background): “Moreover, several impossibility results exist: for instance, equalized odds is incompatible with other formulations of fairness [7, 10, 27], and it is impossible to achieve equalized odds using a calibrated classifier without withholding a randomly selected subset of the classifier’s predictions [35”
- Social Bias Frames: Reasoning about Social and Power Implications of Language (2019, background): “When deploying such algorithms, ethical aspects should be considered including which performance metric should be optimized (Corbett-Davies et al., 2017), as well as the fairness of the model on speech by different demographic groups or in different varieties of English (Mitchell et al.”
- Inherent Trade-Offs in the Fair Determination of Risk Scores (2016, background): “[7] (and see also [6]) consider binary prediction subject to analogues of the balance conditions for the negative and positive classes, together with a form of calibration adapted to binary prediction.”
- Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions (2018, background): “both note that a decision rule δ that maximizes utility under a demographic parity constraint (in general) uses the sensitive variables a both in estimating the conditional probabilities and for determining their thresholds [19, 82].”
- A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual &Group Unfairness via Inequality Indices (2018, background): “, statistical parity [9, 12, 23], disparate impact [13, 41], equality of opportunity [17], calibration [23], and disparate mistreatment [40]).”
- Achieving Fairness through Adversarial Learning: an Application to Recidivism Prediction (2018, background): “Corbett-Davies et al. (2017) argue that because black inmates are arrested at a higher rate than white inmates, there should be some disparity of recidivism prediction for white and black inmates.”
- Deep k-Nearest Neighbors: Towards Confident, Interpretable and Robust Deep Learning (2018, background): “” prediction may have been a consequence of the ball being in the picture. Also note how the white apparel color and general arm positions of players often match the test image of Barack Obama. [98], [99], [100]. This is for instance undesirable when the model’s predictions are used to make decision”
- Fairness Through Computationally-Bounded Awareness (2018, background): “Examples of discrimination in classification have been well-illustrated [24, 1, 5, 19, 13, 3]; nevertheless, developing a systematic approach to fairness has been challenging.”
- Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation (2017, background): “Risk scoring models have a long history of usage in criminal justice, nance, hiring, and other critical domains that impact people’s lives [9, 18].”
- Wasserstein Fair Classification (2019, methodology): “Some methods transform the constrained optimization problem via the method of Lagrange multipliers (Agarwal et al., 2018; Corbett-Davies et al., 2017; Cotter et al., 2018; Goh et al., 2016; Narasimhan, 2018; Wu et al., 2018; Zafar et al., 2017).”
- Proportionally Fair Clustering (2019, background): “Motivated by concerns ranging from the granting of bail in the legal system to the quality of recommender systems, researchers have devoted considerable effort to developing fair algorithms for the canonical supervised learning tasks of classification and regression (Dwork et al., 2012; Kleinberg et”
- Learning Optimal and Fair Decision Trees for Non-Discriminative Decision-Making (2019, methodology): “…, where we predict if an individual earns more than $50k per year and the protected attribute is race; (C) The COMPAS dataset (Angwin et al. 2016; Corbett-Davies et al. 2017) with |N| = 10 , 500 data points and d = 16 , where we predict if a convicted individual will commit a violent crime and…”
- Stable and Fair Classification (2019, methodology): “This approach is applied in several prior work, e.g., [41, 18, 32].”
- 50 Years of Test (Un)fairness: Lessons for Machine Learning (2018, background): “’s framing of the cost of fairness [17].”