A Broader View on Bias in Automated Decision-Making: Reflecting on Epistemology and Dynamics

By Roel Dobbe, Sarah Dean, T. Gilbert, Nitin Kohli. 2018

47 citations recorded by the source corpus.

Machine learning (ML) is increasingly deployed in real world contexts, supplying actionable insights and forming the basis of automated decision-making systems. While issues resulting from biases pre-existing in training data have been at the center of the fairness debate, these systems are also affected by technical and emergent biases, which often arise as context-specific artifacts of implementation. This position paper interprets technical bias as an epistemological problem and emergent bias as a dynamical feedback phenomenon. In order to stimulate debate on how to change machine learning practice to effectively address these issues, we explore this broader view on bias, stress the need to reflect on epistemology, and point to value-sensitive design methodologies to revisit the design and implementation process of automated decision-making systems.

Subfields: Artificial Intelligence, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

Citations in the corpus

Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions (2018, background): “Several recent papers have raised concerns about prediction-based decision systems, describing “biases” or “traps” beyond the focus of most quantitative literature [49, 50, 96, 32, 115].”
Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing (2020, background): “This can shed light on how our position in society biases our assumptions and ways of knowing [18].”
What We Can't Measure, We Can't Understand: Challenges to Demographic Data Procurement in the Pursuit of Fairness (2020, background): “the thinking about how the data should be used, it is both more likely that data will be employed in the interest of those it is about and that the fairness achieved will have material benefits for their lives [22, 29, 47, 54, 57, 77].”
A Framework for Understanding Unintended Consequences of Machine Learning (2019, background): “Similarly, Dobbe et al. (2018) and Mitchell, Potash, and Barocas (2018) walk through choices made when deﬁning a machine learning problem, illustrating how these carry inherent assumptions and value judgments.”
Algorithmic Fairness: Choices, Assumptions, and Definitions (2021, background): “Several recent papers have demonstrated how policy goals are, sometimes clumsily, abstracted and formulated to fit into a prediction task (Dobbe et al. 2018, Eckhouse et al. 2018, Green 2018, Green & Hu 2018, Ochigame et al. 2018, Silva & Kenney 2018, Passi & Barocas 2019, Selbst et al. 2019).”
Pretrained AI Models: Performativity, Mobility, and Change (2019, background): “That is, as noted in classical economic theories of discrimination [4, 17, 84] and recent models of model retraining [22, 33, 67], populations might become more like what (biased) models predict.”