A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle

By Harini Suresh, John V. Guttag. Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, 2019

647 citations recorded by the source corpus.

As machine learning (ML) increasingly affects people and society, awareness of its potential unwanted consequences has also grown. To anticipate, prevent, and mitigate undesirable downstream consequences, it is critical that we understand when and how harm might be introduced throughout the ML life cycle. In this paper, we provide a framework that identifies seven distinct potential sources of downstream harm in machine learning, spanning data collection, development, and deployment. In doing so, we aim to facilitate more productive and precise communication around these issues, as well as more direct, application-grounded ways to mitigate them.

Subfields: Artificial Intelligence, Health Informatics, Safety Research

View Paper: https://www.semanticscholar.org/paper/f4bc6b284bc4508856ddc10d60b30635de9cfb7f

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Health Informatics

In Safety Research

Citations in the corpus

Benchmarking Intersectional Biases in NLP (2022, background): “First, much of the current work on examining NLP bias (and proposing debiasing strategies) has focused on representational harm - how a model describes certain groups, including stereo-typing and other misrepresentations (Blodgett et al., 2020; Suresh and Guttag, 2019).”
GPT-4o System Card (2024, background): “This can lead to both allocative and representational harms [13, 15] depending on how such behavior manifests.”
Collecting a Large-Scale Gender Bias Dataset for Coreference Resolution and Machine Translation (2021, background): “(Suresh and Guttag, 2021). values indicate that the model performs better on stereotypical gender role assignments.”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “A subset of label errors are due to more conceptually consistent disagreements between annotators [27] or dataset bias [146]; these types of errors are more appropriately construed as external validity issues, and are described further in Section 4.4.”
Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction (2022, background): “Design choices [143] and training data [212] influence which people and experiences are legible to an algorithmic system.”
Sociotechnical Harms: Scoping a Taxonomy for Harm Reduction (2022, background): “This process involves design decisions predicated on “selection, reduction, and categorization” [33] through which technologies come to reﬂect the values of certain worldviews [33, 188].”
Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (2022, background): “When individuals feel that they are not being fairly judged when applying for jobs [2, 3, 5, 7, 54–57] or loans [58–60] it can reduce public trust in AI technology [61, 62].”