On Fairness and Calibration

By Geoff Pleiss, Manish Raghavan, Felix Wu, Jon M. Kleinberg, Kilian Q. Weinberger. 2017

1,018 citations recorded by the source corpus.

The machine learning community has become increasingly concerned with the potential for bias and discrimination in predictive models. This has motivated a growing line of work on what it means for a classification procedure to be "fair." In this paper, we investigate the tension between minimizing error disparity across different population groups while maintaining calibrated probability estimates. We show that calibration is compatible only with a single error constraint (i.e. equal false-negatives rates across groups), and show that any algorithm that satisfies this relaxation is no better than randomizing a percentage of predictions for an existing classifier. These unsettling findings, which extend and generalize existing results, are empirically confirmed on several datasets.

Subfields: Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

Citations in the corpus

Measurement and Fairness (2019, background): “r operationalization, i.e., calibration. The fairness in machine learning community responded, largely re ecting the incompatibilities and shortcomings of dierent fairness operationalizations (e.g., [8, 11, 15, 13, 39, 54]). Meanwhile, Chouldechova [11] and Kleinberg et al. [39] proved that the two ”
Does Object Recognition Work for Everyone? (2019, background): “Moreover, several impossibility results exist: for instance, equalized odds is incompatible with other formulations of fairness [7, 10, 27], and it is impossible to achieve equalized odds using a calibrated classifier without withholding a randomly selected subset of the classifier’s predictions [35”
Towards fairer datasets: filtering and balancing the distribution of the people subtree in the ImageNet hierarchy (2019, background): “Researchers have proposed multiple fairness metrics including statistical parity [12, 13, 24, 39], disparate impact [27, 80], equalized odds [34] and individual fairness [23], and analyzed the relationship between them [43, 60].”
The Right Tool for the Job: Matching Model and Instance Complexities (2020, background): “late to existing concerns regarding fairness when using calibrated classifiers (Pleiss et al., 2017).”
Mitigating Gender Bias in Natural Language Processing: Literature Review (2019, background): “…to NLP; other ﬁelds in computer science such as data mining, machine learning, and security also study gender bias (Calders and Verwer, 2010; Feldman et al., 2015; Hardt et al., 2016; Misra et al., 2016; Kleinberg et al., 2016; Pleiss et al., 2017; Beutel et al., 2017; Kilbertus et al., 2017).”
A Survey on Bias and Fairness in Machine Learning (2019, methodology): “Data pre-processing [63] ✓ Classification [131] ✓ Regression [13] ✓ Classification [65] ✓ Classification [128] ✓ Adversarial learning [76] ✓ Classification [55] ✓ Word embedding [19] ✓ Classification [103] ✓ Table 3. Algorithms categorized into their appropriate groups based on being pre-processing,”
Learning Adversarially Fair and Transferable Representations (2018, background): “Formally, this is Satisfying these constraints is known to conﬂict with learning well-calibrated classiﬁers (Chouldechova, 2017; Klein-berg et al., 2016; Pleiss et al., 2017).”
Fairness Through Computationally-Bounded Awareness (2018, background): “As has been well-documented [19, 4, 21], calibration and equalized error rates, in general, cannot be simultaneously satisfied.”
Statistical Equity: A Fairness Classification Objective (2020, methodology): “Other deﬁnitions require post-processing techniques in which discrimination removal is performed after the training phase treating the model as a black box [Hardt et al., 2016, Pleiss et al., 2017].”
Flexibly Fair Representation Learning by Disentanglement (2019, background): “First is the question of fairness metrics: a wide range of fairness metrics beyond demographic parity have been proposed (Hardt et al., 2016; Pleiss et al., 2017).”
Proportionally Fair Clustering (2019, background): “Motivated by concerns ranging from the granting of bail in the legal system to the quality of recommender systems, researchers have devoted considerable effort to developing fair algorithms for the canonical supervised learning tasks of classification and regression (Dwork et al., 2012; Kleinberg et”
Stable and Fair Classification (2019, background): “Another approach for fair classiﬁcation is to shift the decision boundary of a baseline classiﬁer, e.g., [28, 37, 34, 61, 73, 22].”
Taking Advantage of Multitask Learning for Fair Classification (2018, background): “This is in contrast to other approaches which impose the fairness constraint as a post-processing step [10 19 20 31] or by modifying the data representation before employing standard machine learning methods [1 12 22–24 39].”
AI Fairness 360: An Extensible Toolkit for Detecting, Understanding, and Mitigating Unwanted Algorithmic Bias (2018, background): “Calibrated equalized odds post-processing (Pleiss et al., 2017) optimizes over calibrated classiﬁer score outputs to ﬁnd probabilities with which to change output labels with an equalized odds objective.”
Delayed Impact of Fair Machine Learning (2018, background): “These trade-offs carry over to some extent to the case where we only equalize true positive rates (Pleiss et al., 2017).”
Fairness in Criminal Justice Risk Assessments: The State of the Art (2017, background): “Arguably, the trade-off that has gotten the most attention is between conditional use accuracy equality and the false positive and false negative rates (Angwin et al. 2016; Chouldechova 2017; Dieterich et al. 2016; Kleinberg et al. 2016; Pleiss et al. 2017).”
Characterizing Fairness Over the Set of Good Models Under Selective Labels (2021, neutral): “Post-processing techniques typically modify the predictions from an existing model to achieve a target notion of fairness (Hardt et al., 2016; Pleiss et al., 2017; Kim et al., 2019).”
Learning Optimized Risk Scores (2016, background): “, 2005), prediction (Reilly and Evans, 2006) and calibration (Pleiss et al., 2017).”