Certifying and Removing Disparate Impact

By Michael Feldman, Sorelle A. Friedler, John Moeller, Carlos Eduardo Scheidegger, Suresh Venkatasubramanian. 2014

2,286 citations recorded by the source corpus.

What does it mean for an algorithm to be biased? In U.S. law, unintentional bias is encoded via disparate impact, which occurs when a selection process has widely different outcomes for different groups, even as it appears to be neutral. This legal determination hinges on a definition of a protected class (ethnicity, gender) and an explicit description of the process. When computers are involved, determining disparate impact (and hence bias) is harder. It might not be possible to disclose the process. In addition, even if the process is open, it might be hard to elucidate in a legal setting how the algorithm makes its decisions. Instead of requiring access to the process, we propose making inferences based on the data it uses. We present four contributions. First, we link disparate impact to a measure of classification accuracy that while known, has received relatively little attention. Second, we propose a test for disparate impact based on how well the protected class can be predicted from the other attributes. Third, we describe methods by which data might be made unbiased. Finally, we present empirical evidence supporting the effectiveness of our test for disparate impact and our approach for both masking bias and preserving relevant information in the data. Interestingly, our approach resembles some actual selection practices that have recently received legal scrutiny.

Subfields: Artificial Intelligence, Management of Technology and Innovation, Safety Research, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management of Technology and Innovation

In Safety Research

In Safety, Robustness & Interpretability

Citations in the corpus

Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (2016, background): “[10] distinguish classification algorithms that achieve fairness by modifying the underlying data from those that achieve fairness by modifying the classification algorithm.”
Does Object Recognition Work for Everyone? (2019, background): “Because statistical parity may be inappropriate when base rates differ between groups, disparate impact [15, 47] poses that positiveclassification rates between any two groups should not vary by more than 80%.”
Towards fairer datasets: filtering and balancing the distribution of the people subtree in the ImageNet hierarchy (2019, background): “Researchers have proposed multiple fairness metrics including statistical parity [12, 13, 24, 39], disparate impact [27, 80], equalized odds [34] and individual fairness [23], and analyzed the relationship between them [43, 60].”
Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints (2017, background): “e cost of making mistakes on samples in the minority class can be neglected. Various approaches have been proposed to make a “fair” binary classiﬁcation (Barocas and Selbst, 2014; Dwork et al., 2012; Feldman et al., 2015; Zliobaite, 2015). For structured prediction tasks the effect is harder to quan”
Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods (2018, background): “Various other approaches have been proposed to produce “fair” classifiers (Calders et al., 2009; Feldman et al., 2015; Misra et al., 2016).”
Mitigating Gender Bias in Natural Language Processing: Literature Review (2019, background): “…discussed extensively in artiﬁcial intelligence, machine learning, data mining, and several other application ﬁelds (e.g., (Calders and Verwer, 2010; Feldman et al., 2015; Hardt et al., 2016; Misra et al., 2016; Kleinberg et al., 2016; Pleiss et al., 2017; Beutel et al., 2017; Misra et al., 2016)).”
Inherent Trade-Offs in the Fair Determination of Risk Scores (2016, background): “This latter global requirement is a version of statistical parity [12, 4, 21, 22].”
A Survey on Bias and Fairness in Machine Learning (2019, background): “Other pre-processing approaches, such as messaging [74], preferential sampling [75, 76], disparate impact removal [51], also aim to remove biases from the data.”
Fair Resource Allocation in Federated Learning (2019, methodology): “In addition to preprocess the data (Feldman et al., 2015) and post-process the model (Feldman,”
Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions (2018, methodology): “[65, 11, 36, 61] give algorithms to build models achieving this notion of fairness”
Fairness Without Demographics in Repeated Loss Minimization (2018, background): “, 2016), disparate impact (Feldman et al., 2015) and calibration (Kleinberg et al.”
On the Art and Science of Machine Learning Explanations (2018, methodology): “o fairness and model debugging techniques such as: •Disparate impact analysis and remediation: to uncover and remediate any disparate impact in model predictions or errors across demographic segments [7]. •Residual analysis: to check the fundamental assumptions of the model against relevant data par”
A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual &Group Unfairness via Inequality Indices (2018, background): “, statistical parity [9, 12, 23], disparate impact [13, 41], equality of opportunity [17], calibration [23], and disparate mistreatment [40]).”
Distill-and-Compare: Auditing Black-Box Models Using Transparent Model Distillation (2017, methodology): “On the other hand, one could study the actual outcome y O , testing for disparate impact through methods such as training a model to predict y O , removing, permuting, or obscuring a protected feature [1, 11], then retraining the model.”
Conscientious Classification: A Data Scientist's Guide to Discrimination-Aware Classification (2017, neutral): “29 This is in spite of the fact that, as can be seen in the empirical studies we cite, 12,17,27 differences between the highest performing state of the art and previously established methods are often within a few percentage points of each other.”