Bias in computer systems

By Batya Friedman, Helen Nissenbaum. 1996

Subfields: Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Safety, Robustness & Interpretability

Citations in the corpus

Measurement and Fairness (2019, background): “ld ever be used, regardless of how well the construct is operationalized|that is, consequential validity considers the potential downstream societal impacts. Analogous to considering values in design [24], Messick argued that the use and consequences of a ‘test’|i.e., a measurement model embedded in”
Toward Gender-Inclusive Coreference Resolution (2019, background): “This can lead to significant biases in coreference resolution systems: cases where systems “systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others” [Friedman and Nissenbaum 1996, p. 332].”
Diagnosing Gender Bias in Image Recognition Systems (2020, background): “At the same time, recent research has shown algorithmic classification systems to be mechanisms for the reproduction, and even amplification, of more general social biases (Friedman and Nissenbaum 1996; Noble 2018).”
Data Statements for Natural Language Processing: Toward Mitigating System Bias and Enabling Better Science (2018, methodology): “Friedman and Nissenbaum (1996) show that in some cases, system bias reﬂects biases in society; these are pre-existing biases with roots in social institutions, practices and attitudes.”
Explanation in Human-AI Systems: A Literature Meta-Review, Synopsis of Key Ideas and Publications, and Bibliography for Explainable AI (2019, neutral): “…Greathead, and Baxter, 2004; Byrne, 2002; Chunpir and Ludwig, 2017; De Kleer and Brown, 2014; Erlich, 1996; Friedman, forbus, and Sherin, 2018; Friedman and Nissenbaum, 1996; Gary and Wood, 2011; Gentner and Stevens, 1983; Gigerenzer, 1993; Gigerenzer, Hoffrage, and Kleinbölting, 1991;…”
Social Data: Biases, Methodological Pitfalls, and Ethical Boundaries (2019, background): “Broadly discussing bias in computer systems, Friedman and Nissenbaum (1996) characterize it according to its source such as societal, technical, or usage related.”
Does mitigating ML's disparate impact require disparate treatment? (2017, background): “These issues have come to light starting with the ground-breaking paper [FN96], which recognized the ability of computer systems to disparately affect protected groups.”
Improving Fairness in Machine Learning Systems: What Do Industry Practitioners Need? (2018, background): “The design, prototyping, and maintenance of machine learning systems raises many unique challenges [27, 63, 78, 87] not commonly faced with other kinds of intelligent systems or computing systems more broadly [34, 66, 67].”
Problem Formulation and Fairness (2019, background): “into the system during its development and implementation [17].”
Addressing Age-Related Bias in Sentiment Analysis (2018, background): “Much work in critical algorithm studies examines algorithmic bias, which can be defined as “systems that systematically and unfairly discriminate against certain individuals or groups of individuals in favor of others” [27].”
Bias in algorithmic filtering and personalization (2013, background): “Just as any computer system, they can unfairly discriminate against certain individuals or groups of individuals in favor of others (Friedman and Nissenbaum 1996).”
Toward Annotator Group Bias in Crowdsourcing (2021, unknown): “Bias is caused due to pre-existing societal norms (Friedman and Nissenbaum, 1996), data source, data labeling, training algorithms, and post-processing models.”
Co-Designing Checklists to Understand Organizational Challenges and Opportunities around Fairness in AI (2020, methodology): “Although HCI researchers have developed methods for designing systems with human values in mind (e.g., [38,39,91,93]), they have not yet become best practices or even propagated into professional training for UX researchers.”
Where fairness fails: data, algorithms, and the limits of antidiscrimination discourse (2019, background): “In these cases, biases are said to ‘ sneak in ’ (University of Bath, 2017), ‘ whether on purpose or by accident ’ (Barocas & Selbst, 2016, p. 674), or in ways that only emerge over time (Friedman & Nissenbaum, 1996).”
Bridging the Gap Between Ethics and Practice (2020, background): “An early review by Friedman and Nissenbaum [39] described three kind of bias, which remain as a helpful guide: preexisting biases based on social practices and attitudes, technical bias based on design constraints in hardware and software, and emergent bias that arises from changing the use context.”