Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse

By Anna Lauren Hoffmann. 2019

407 citations recorded by the source corpus.

ABSTRACT Problems of bias and fairness are central to data justice, as they speak directly to the threat that ‘big data’ and algorithmic decision-making may worsen already existing injustices. In the United States, grappling with these problems has found clearest expression through liberal discourses of rights, due process, and antidiscrimination. Work in this area, however, has tended to overlook certain established limits of antidiscrimination discourses for bringing about the change demanded by social justice. In this paper, I engage three of these limits: 1) an overemphasis on discrete ‘bad actors’, 2) single-axis thinking that centers disadvantage, and 3) an inordinate focus on a limited set of goods. I show that, in mirroring some of antidiscrimination discourse’s most problematic tendencies, efforts to achieve fairness and combat algorithmic discrimination fail to address the very hierarchical logic that produces advantaged and disadvantaged subjects in the first place. Finally, I conclude by sketching three paths for future work to better account for the structural conditions against which we come to understand problems of data and unjust discrimination in the first place.

Subfields: Safety Research, Sociology and Political Science, AI Ethics, Fairness & Society, Social Sciences & Humanities

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety Research

In Sociology and Political Science

In AI Ethics, Fairness & Society

In Social Sciences & Humanities

Citations in the corpus

Towards a critical race methodology in algorithmic fairness (2019, neutral): “rent algorithmic fairness frameworks [53, 108].”
Smarter eco-cities and their leading-edge artificial intelligence of things solutions for environmental sustainability: A comprehensive systematic review (2023, background): “The challenges posed by AI generally involve gaps in ethical standards, including safety, fairness, transparency [293,296], socio-economic equality, cultural diversity, and social inclusion.”
How Platform-User Power Relations Shape Algorithmic Accountability: A Case Study of Instant Loan Platforms and Financially Stressed Users in India (2022, background): “While doing so, we must also recognize that a purely computation framing of harms fails to address injustices caused by structural oppression [63, 77].”
Algorithmic content moderation: Technical and political challenges in the automation of platform governance (2020, background): “Fairness critiques often miss broader structural issues, and risk being blind to wider patterns of systemic harm (Hoffmann, 2019).”
Algorithmic realism: expanding the boundaries of algorithmic thought (2020, background): “Even within the FAT* community, critical evaluations of the mathematization of fairness suggest the need for further evolution from formalism towards realism [11, 52, 58, 67, 135].”
Discriminated by an algorithm: a systematic review of discrimination and fairness by algorithmic decision-making in the context of HR recruitment and HR development (2020, background): “…review (Lee 2018; Tambe et al. 2019; Yarger et al. 2019), two articles were excluded in the eligibility stage of the initial search process (Hoffmann 2019; Sumser 2017) (no reference to HRM and comment), and the remaining three articles neither discussed fairness nor the HR recruitment…”
On the Legal Compatibility of Fairness Definitions (2019, background): “s to categorize and stereotype individuals in a way that masks more speciﬁc individual-level harms, at best only considering additive combinations of unitary classes (i.e., you are white and a man) 3 [13; 23]. Evenattemptsat moregranularalgorithmic“individualfairness” reliesona similarity metric and”
“It’s Complicated”: Negotiating Accessibility and (Mis)Representation in Image Descriptions of Race, Gender, and Disability (2021, background): “For marginalized groups, the misclassification of identity is a form of violence and reflective of greater societal discrimination [11, 17, 50, 53, 54, 64, 99].”
What is the point of fairness? (2019, background): “that a lack of fairness is a failure of implementation, and largely center increasing the representativeness of the data underlying the algorithm, and improving how the algorithm integrates “outliers”[29, 12]. An initial reading of fairness may make it seem a reasonable goal to center in algorithmic”
Fairness is not static: deeper understanding of long term fairness via simulation studies (2020, background): “More recently, a number of critical evaluations of the algorithmic fairness literature draw attention to the ways that machine learning systems interact with, and often reinforce, the power structures that generate the inequities that this literature is meant to address [3, 15, 28].”
Terms of inclusion: Data, discourse, violence (2020, background): “Indicative of a larger body of limited and atomistic responses to problems of structural violence and data discrimination (see Hoffmann, 2019), inclusion discourses work as a convenient cover for problems that are—as Oscar Gandy (1993) demonstrated—not separate from, but intrinsic to technologies…”
Interactive Model Cards: A Human-Centered Approach to Model Documentation (2022, background): “The potential for both broad and deep harm that AI/ML systems can pose to vulnerable people and ecosystems, especially when impacted people do not have the opportunity to interrogate and contest decisions driven by AI/ML(e.g., [4, 25, 26, 29, 72]).”
Confronting Power and Corporate Capture at the FAccT Conference (2022, background): “FAccT must also respond to contemporary and historical struggles against oppression by engaging in critical reflection and by re-centering its organizing principles and work [15, 17, 34, 70] to-”