A Framework for Understanding Unintended Consequences of Machine Learning

By Harini Suresh, John V. Guttag. 2019

416 citations recorded by the source corpus.

Subfields: AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In AI Ethics, Fairness & Society

Citations in the corpus

A Survey on Bias and Fairness in Machine Learning (2019, methodology): “An example of this type of bias can be found in a 2018 image search result where searching for women CEOs ultimately resulted in fewer female CEO images due to the fact that only 5% of Fortune 500 CEOs were woman—which would cause the search results to be biased towards male CEOs [117].”
Prediction-Based Decisions and Fairness: A Catalogue of Choices, Assumptions, and Definitions (2018, neutral): “a non-statistical notion of societal bias [118].”
The Values Encoded in Machine Learning Research (2021, background): “Although a plethora of work exists on sources of harm that can arise in relation to ML research [7, 15, 25, 28, 60], we observe that these discussions are ignored in these influential conference publications.”
Five sources of bias in natural language processing (2021, background): “Suresh and Guttag (2019) suggested a qualitative framework for bias in machine learning, defining bias as a ‘potential harmful property of the data’, though they leave out demographic and modelling aspects.”
Discriminated by an algorithm: a systematic review of discrimination and fairness by algorithmic decision-making in the context of HR recruitment and HR development (2020, background): “For example, if the input data of an algorithm include implicit or explicit human judgments, stereotypes, or biases, an accurate algorithmic output will inevitably entail these human judgments, stereotypes, and prejudices (Diakopoulos 2015; Suresh and Guttag 2019; Barfield and Pagallo 2018).”
Algorithmic Fairness: Choices, Assumptions, and Definitions (2021, background): “This corresponds to a nonstatistical notion of societal bias (Suresh & Guttag 2019).”
Gender Bias in Machine Translation (2021, background): “Treating data as an abstract, monolithic entity (Gitelman, 2013)—or relying on ‘‘overly broad/overloaded terms like training data bias’’8 (Suresh and Guttag, 2019)—do not encourage reasoning on the many factors of which data are the product: first and foremost, the historical, socio-cultural context”
Handling Bias in Toxic Speech Detection: A Survey (2022, methodology): “Each bias mitigation method can be applied to one or more data transformation stages of a machine learning (ML) pipeline [82].”
Sex and gender differences and biases in artificial intelligence for biomedicine and healthcare (2020, background): “Ethical standards will have to continue to be considered by governments and regulatory organisations to guarantee the preservation of personal data privacy and security as well as to determine the way new technological tools should be employed, data should be collected, and models improved 121,122 .”
Expanding Explainability: Towards Social Transparency in AI systems (2021, background): “Otherwise one may risk deploying an AI system un-integrated into individual and organizational workflows [66, 106], potentially resulting in misuse, mistrust [108, 109], or having profound ethical risks and unintended consequences, especially for marginalized groups [72, 86, 99].”
Say No to the Discrimination: Learning Fair Graph Neural Networks with Limited Sensitive Attribute Information (2020, background): “The bias would largely limit the wide adoption of GNNs in domains such as ranking of job applicants [32] and crime rate prediction [40].”
Lessons from archives: strategies for collecting sociocultural data in machine learning (2019, background): “Even before collection, data are subject to two levels of bias - historical and representational [60].”
Bias in machine learning - what is it good for? (2020, background): “His-torical bias has to do with data that in itself has unwanted properties that are regarded as biased: ‘Historical bias arises even if the data is perfectly measured and sampled, if the world as it is or was leads a model to produce outcomes that are not wanted’ [45].”
Fairness in Machine Learning: A Survey (2020, background): “” Several publications provide a (limited) overview of multiple fairness metrics and definitions, e.g., References [108, 188, 238, 245, 275, 288].”
How We've Taught Algorithms to See Identity: Constructing Race and Gender in Image Databases for Facial Analysis (2020, background): “Through the work of Stuart Hall, we see these as moments of identification , the situated construction of identity categories, to include or exclude specific groups of people [61].”
Towards a Standard for Identifying and Managing Bias in Artificial Intelligence (2022, background): “The set of harms that can arise from the use of NLP however has become a recent concern in the area of trustworthy AI [81, 91, 158, 168, 169].”