Big Data's Disparate Impact
By Solon Barocas, Andrew D. Selbst. 2016
3,122 citations recorded by the source corpus.
Advocates of algorithmic techniques like data mining argue that these techniques eliminate human biases from the decision-making process. But an algorithm is only as good as the data it works with. Data is frequently imperfect in ways that allow these algorithms to inherit the prejudices of prior decision makers. In other cases, data may simply reflect the widespread biases that persist in society at large. In still others, data mining can discover surprisingly useful regularities that are really just preexisting patterns of exclusion and inequality. Unthinking reliance on data mining can deny historically disadvantaged and vulnerable groups full participation in society. Worse still, because the resulting discrimination is almost always an unintentional emergent property of the algorithm’s use rather than a conscious choice by its programmers, it can be unusually hard to identify the source of the problem or to explain it to a court.This Essay examines these concerns through the lens of American antidiscrimination law — more particularly, through Title VII’s prohibition of discrimination in employment. In the absence of a demonstrable intent to discriminate, the best doctrinal hope for data mining’s victims would seem to lie in disparate impact doctrine. Case law and the Equal Employment Opportunity Commission’s Uniform Guidelines, though, hold that a practice can be justified as a business necessity when its outcomes are predictive of future employment outcomes, and data mining is specifically designed to find such statistical correlations. Unless there is a reasonably practical way to demonstrate that these discoveries are spurious, Title VII would appear to bless its use, even though the correlations it discovers will often reflect historic patterns of prejudice, others’ discrimination against members of protected groups, or flaws in the underlying dataAddressing the sources of this unintentional discrimination and remedying the corresponding deficiencies in the law will be difficult technically, difficult legally, and difficult politically. There are a number of practical limits to what can be accomplished computationally. For example, when discrimination occurs because the data being mined is itself a result of past intentional discrimination, there is frequently no obvious method to adjust historical data to rid it of this taint. Corrective measures that alter the results of the data mining after it is complete would tread on legally and politically disputed terrain. These challenges for reform throw into stark relief the tension between the two major theories underlying antidiscrimination law: anticlassification and antisubordination. Finding a solution to big data’s disparate impact will require more than best efforts to stamp out prejudice and bias; it will require a wholesale reexamination of the meanings of “discrimination” and “fairness.”
Subfields: Information Systems, Information Systems and Management, Safety Research, AI Ethics, Fairness & Society, Social Sciences & Humanities
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Information Systems
- Emilio Ferrara
- Filippo Menczer
- Onur Varol
- Kai-Cheng Yang
- Carlos Castillo
- Nathan Kallus
- Sara Hajian
- Alessandro Flammini
In Information Systems and Management
- Nripendra P. Rana
- Yogesh Kumar Dwivedi
- Kuttimani Tamilmani
- Ali Abdallah Alalwan
- Rohita Dwivedi
- Arpan Kumar Kar
- Michael D. Williams
- Hatice Kizgin
In Safety Research
- Silvia Chiappa
- Dana Pessach
- Erez Shmueli
- Emilio Ferrara
- Luca Oneto
- Xintao Wu
- Lu Zhang
- Indrė Žliobaitė
In AI Ethics, Fairness & Society
- Silvia Chiappa
- Dana Pessach
- Erez Shmueli
- Xintao Wu
- Indrė Žliobaitė
- Lu Zhang
- Luca Oneto
- Adrian Weller
In Social Sciences & Humanities
- Manuel Gomez-Rodriguez
- Krishna P. Gummadi
- Adrian Weller
- Reuben Binns
- Adam Tauman Kalai
- Ben Green
- Muhammad Bilal Zafar
- Angèle Christin
Citations in the corpus
- Gender Bias in Coreference Resolution (2018, background): “As awareness grows of the ways in which data-driven AI technologies may acquire and amplify human-like biases (Caliskan et al., 2017; Barocas and Selbst, 2016; Hovy and Spruit, 2016), this work investigates how gender biases manifest in coreference resolution systems.”
- Ethical and social risks of harm from Language Models (2021, background): “Various steps in training a machine learning model can result in discrimination in the sociotechnical sense, from labelling and collection of the training data, to defining the “target variable” and class labels, to selecting features (Barocas and Selbst, 2016).”
- Measurement and Fairness (2019, background): “urces of harm, interrogate our values, and question our assumptions. For example, by recognizing that risk scores are inferred from arrest data, which re ects policing practices, not crimes committed [6, 43], we are better able to assess their consequential validity, potentially allowing us to concl”
- Exposing and Correcting the Gender Bias in Image Captioning Datasets and Models (2019, methodology): “With the advent of big-data, the bias has made its way into the datasets [Barocas and Selbst, 2016, Data et al., 2004, Torralba et al., 2011, Beigman and Klebanov, 2009].”
- Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings (2016, background): “A definition of fairness based on legal traditions is presented by Barocas and Selbst [3].”
- Towards Understanding and Mitigating Social Biases in Language Models (2021, background): “Further usages of such approaches can amplify biases and unfairly discriminate against users, particularly those from disadvantaged social groups (Barocas and Selbst, 2016; Sun et al., 2019; In this paper, we aim to provide a more formal understanding of social biases in LMs.”
- Semantics derived automatically from language corpora contain human-like biases (2016, background): “Besides, bias is known to creep in indirectly, by proxy (Barocas and Selbst, 2014).”
- Towards the Science of Security and Privacy in Machine Learning (2016, background): “Historical data also inherently reflects social biases [108].”
- Women also Snowboard: Overcoming Bias in Captioning Models (2018, background): “, gender, ethnic biases) have been studied across a wide variety of AI domains [5,6,7,8,9,10].”
- Measuring Bias in Contextualized Word Representations (2019, background): “…2018) and language identification (Jurgens et al., 2017), to downstream systems such as automated essay scoring (Amorim et al., 2018), exhibit inherent social biases which are attributed to the datasets used to train the embeddings (Barocas and Selbst, 2016; Zhao et al., 2017; Yao and Huang, 2017).”
- Towards fairer datasets: filtering and balancing the distribution of the people subtree in the ImageNet hierarchy (2019, background): “In short, datasets often have biased distributions of demographics (gender, race, age, etc.); machine learning models are trained to exploit whatever correlations exist in the data, leading to discriminatory behavior against underrepresented groups [6, 7].”
- Social Bias in Elicited Natural Language Inferences (2017, background): “High-profile cases of overgener-alization in the public sphere abound (Crawford, 2013; Crawford, 2016; Barocas and Selbst, 2016).”
- Black is to Criminal as Caucasian is to Police: Detecting and Removing Multiclass Bias in Word Embeddings (2019, background): “Machine learning systems that use embeddings can further amplify biases (Barocas and Selbst, 2016; Zhao et al., 2017), discriminating against users, particularly those from disadvantaged social groups.”
- Certifying and Removing Disparate Impact (2014, background): “Instead, we direct the reader to the survey of Romei and Ruggieri [18] and to a discussion of the issues specific to data mining and disparate impact [1].”
- Inherent Trade-Offs in the Fair Determination of Risk Scores (2016, background): “Barocas and Selbst survey a range of ways in which data-analysis algorithms can lead to discriminatory outcomes [3], and review articles by Romei and Ruggieri [25] and Zliobaite [30] survey data-analytic and algorithmic methods for measuring discrimination.”