Residual Unfairness in Fair Machine Learning from Prejudiced Data

By Nathan Kallus, Angela Zhou. 2018

144 citations recorded by the source corpus.

Recent work in fairness in machine learning has proposed adjusting for fairness by equalizing accuracy metrics across groups and has also studied how datasets affected by historical prejudices may lead to unfair decision policies. We connect these lines of work and study the residual unfairness that arises when a fairness-adjusted predictor is not actually fair on the target population due to systematic censoring of training data by existing biased policies. This scenario is particularly common in the same applications where fairness is a concern. We characterize theoretically the impact of such censoring on standard fairness metrics for binary classifiers and provide criteria for when residual unfairness may or may not appear. We prove that, under certain conditions, fairness-adjusted classifiers will in fact induce residual unfairness that perpetuates the same injustices, against the same groups, that biased the data to begin with, thus showing that even state-of-the-art fair machine learning can have a "bias in, bias out" property. When certain benchmark data is available, we show how sample reweighting can estimate and adjust fairness metrics while accounting for censoring. We use this to study the case of Stop, Question, and Frisk (SQF) and demonstrate that attempting to adjust for fairness perpetuates the same injustices that the policy is infamous for.

Subfields: Safety Research, Sociology and Political Science, Statistics and Probability, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety Research

In Sociology and Political Science

In Statistics and Probability

In Safety, Robustness & Interpretability

Citations in the corpus

Re-imagining Algorithmic Fairness in India and Beyond (2021, background): “Entire communities may be missing or misrepresented in datasets, exacerbated by digital divides, leading to wrong conclusions [30, 52, 53, 119] and residual unfairness [103].”
Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions (2018, background): “probability of crime given covariates), biased estimation of utility, and inadequate fairness adjustments [63].”
Characterizing Fairness Over the Set of Good Models Under Selective Labels (2021, background): “However, evaluating a model on a population different than the one on which it will be used can be highly misleading, particularly with regards to predictive fairness measures [35, 36].”
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? (2018, background): “ neglected in the literature so far, as well as several areas of alignment. For example, while the fair ML literature has largely focused on “de-biasing” methods and viewed the training data as fixed [23, 59], most of our interviewees report that their teams consider data collection, rather than mod”
Fairness Violations and Mitigation under Covariate Shift (2019, background): “Our work is related in spirit to [6, 29] who consider building fair models from ‘biased’ training data.”
Counterfactual risk assessments, evaluation, and fairness (2019, background): “Our work bears conceptual similarity to the analysis of residual unfairness when there is selection bias in the training data that induces covariate shift at test time as discussed in [19].”
Why Machine Learning May Lead to Unfairness: Evidence from Risk Assessment for Juvenile Justice in Catalonia (2019, background): “These mechanisms could lead to a bad track record for specific minority groups, creating downwards dynamics in terms of risk assessment [25, 31].”
Fairness in Machine Learning: A Survey (2020, background): “Reference [164] has noted that this dilemma can be articulated as a bias in, bias out property of ML; i.e., addressing one form of bias results in another.”
It's COMPASlicated: The Messy Relationship between RAI Datasets and Algorithmic Fairness Benchmarks (2021, background): “180 [56, 57] study conditions when these distributional differences lead to biases in incomplete fairness 181 adjustment, although normative implications depend on the joint utility associated with censorship 182 in/out of the dataset and the final positive outcome of interest.”
Fair Decisions Despite Imperfect Predictions (2019, methodology): “We build on previous work on counterfactual inference and policy learning (Athey & Wager, 2017; Ensign et al., 2018; Gillen et al., 2018; Heidari & Krause, 2018; Joseph et al., 2016; Jung et al., 2018; Kallus, 2018; Kallus & Zhou, 2018; Lakkaraju & Rudin, 2017).”
Economics, Fairness and Algorithmic Bias (2020, unknown): “A related, smaller literature prescribes new computational techniques for developing algorithms that are ex ante more likely to be unbiased (Lum and Johndrow, 2016; Johndrow and Lum, 2017; Aliverti et al., 2018; Kallus and Zhou, 2018).”
Learning under selective labels in the presence of expert consistency (2018, unknown): “The fairness-related risks of learning from censored data are explored in Kallus & Zhou (2018).”
Algorithmic Fairness: Choices, Assumptions, and Definitions (2021, background): “In the algorithmic fairness literature, data with these undesirable properties are often labeled informally as biased (e.g., Kamiran & Calders 2009, Barocas & Selbst 2016, Chouldechova 2017, Lipton et al. 2018).”