Inherent Trade-Offs in the Fair Determination of Risk Scores
By Jon M. Kleinberg, Sendhil Mullainathan, Manish Raghavan. 2016
2,087 citations recorded by the source corpus.
Recent discussion in the public sphere about algorithmic classification has involved tension between competing notions of what it means for a probabilistic classification to be fair to different groups. We formalize three fairness conditions that lie at the heart of these debates, and we prove that except in highly constrained special cases, there is no method that can satisfy these three conditions simultaneously. Moreover, even satisfying all three conditions approximately requires that the data lie in an approximate version of one of the constrained special cases identified by our theorem. These results suggest some of the ways in which key notions of fairness are incompatible with each other, and hence provide a framework for thinking about the trade-offs between them.
Subfields: Artificial Intelligence, History and Philosophy of Science, Safety, Robustness & Interpretability
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Percy Liang
- Mohit Bansal
- Yoshua Bengio
- Sergey Levine
- Quoc V. Le
- Jianfeng Gao
- Ruslan Salakhutdinov
- Yejin Choi
In History and Philosophy of Science
- Susan Athey
- Aaron Roth
- Ben Green
- Alexandra Chouldechova
- Krishna P. Gummadi
- Reuben Binns
- Michael Kearns
- Adrian Weller
In Safety, Robustness & Interpretability
- Silvia Chiappa
- Dana Pessach
- Erez Shmueli
- Percy Liang
- Adrian Weller
- Emilio Ferrara
- Aaron Roth
- Krishna P. Gummadi
Citations in the corpus
- Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs (2021, background): “For example, well-known impossibility results imply that performance disparities according to one metric may originate from a decision to reduce performance disparities according to another metric [16, 48].”
- Measurement and Fairness (2019, background): “, [8, 11, 15, 13, 39, 54]).”
- Towards the Science of Security and Privacy in Machine Learning (2016, background): “It should not nurture discrimination against specific individuals [107].”
- Does Object Recognition Work for Everyone? (2019, background): “Moreover, several impossibility results exist: for instance, equalized odds is incompatible with other formulations of fairness [7, 10, 27], and it is impossible to achieve equalized odds using a calibrated classifier without withholding a randomly selected subset of the classifier’s predictions [35”
- Equality of Opportunity in Supervised Learning (2016, background): “In a recent concurrent work, Kleinberg, Mullainathan and Raghavan [KMR16] showed that in general a score that is calibrated within each group does not satisfy a criterion equivalent to equalized odds for binary predictors.”
- Mitigating Gender Bias in Natural Language Processing: Literature Review (2019, background): “…to NLP; other fields in computer science such as data mining, machine learning, and security also study gender bias (Calders and Verwer, 2010; Feldman et al., 2015; Hardt et al., 2016; Misra et al., 2016; Kleinberg et al., 2016; Pleiss et al., 2017; Beutel et al., 2017; Kilbertus et al., 2017).”
- Mitigating Unwanted Biases with Adversarial Learning (2018, background): “See also Kleinberg, Mullainathan, and Raghavan [7] for incongruency between calibration and equalized odds. model (since other variables can be highly correlated with the protected variable), and devise a method for learning fair predictive models in cases when the learning model is simple (e.g. lin”
- A Survey on Bias and Fairness in Machine Learning (2019, background): “It is important to note that according to [70], it is impossible to satisfy some of the fairness constraints at once except in highly constrained special cases.”
- Counterfactual Fairness (2017, background): “There has been much recent interest in designing algorithms that make fair predictions [4, 6, 9, 10, 12, 14–17, 19, 21, 30–33].”
- Fairness Without Demographics in Repeated Loss Minimization (2018, background): “There has been extensive work (Barocas & Selbst, 2016) on guaranteeing fairness for classification over a protected label through constraints such as equalized odds (Woodworth et al., 2017; Hardt et al., 2016), disparate impact (Feldman et al., 2015) and calibration (Kleinberg et al., 2017).”
- Learning Adversarially Fair and Transferable Representations (2018, background): “Satisfying these constraints is known to conflict with learning well-calibrated classifiers (Chouldechova, 2017; Kleinberg et al., 2016; Pleiss et al., 2017).”
- A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual &Group Unfairness via Inequality Indices (2018, background): “, statistical parity [9, 12, 23], disparate impact [13, 41], equality of opportunity [17], calibration [23], and disparate mistreatment [40]).”
- Achieving Fairness through Adversarial Learning: an Application to Recidivism Prediction (2018, background): “Hardt 1 et al. (2016) and Kleinberg et al. (2016) provide definitions for these types of fairness and discuss their trade-offs.”