Moving beyond “algorithmic bias is a data problem”

By Sara Hooker. 2021

193 citations recorded by the source corpus.

A surprisingly sticky belief is that a machine learning model merely reflects existing algorithmic bias in the dataset and does not itself contribute to harm. Why, despite clear evidence to the contrary, does the myth of the impartial model still hold allure for so many within our research community? Algorithms are not impartial, and some design choices are better than others. Recognizing how model design impacts harm opens up new mitigation techniques that are less burdensome than comprehensive data collection.

Subfields: Artificial Intelligence, Safety Research, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “The overall level of harm is also impacted by modeling choice [108].”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “Modeling choices (Hooker, 2021) and the lack of control, e.”
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, background): “We also highlight that algorithmic choices can introduce biases on their own [Hooker, 2021]: one limitation of our study is that we did not explore whether speciﬁc architectures and objectives had an impact on the toxicity and biases of a given model.”
Randomness In Neural Network Training: Characterizing The Impact of Tooling (2021, background): “Trade-off with fairness objectives Recent work (Hooker, 2021; Yona et al., 2021; D’Amour et al., 2020; Hooker et al., 2019) has identified that models with similar top-line metrics can evidence unacceptable performance on subsets of the distribution.”
Generative artificial intelligence (2023, background): “Overfitting is a typical phenomenon that originates from the chosen learning strategies or optimization functions and causes biased algorithmic outcomes (Danks & London, 2017; Hooker, 2021).”
A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle (2019, background): “Learning bias arises when modeling choices amplify performance disparities across different examples in the data [26].”
Manipulating SGD with Data Ordering Attacks (2021, background): “Hooker has explained that bias in ML is not just a data problem, but depends on algorithms in subtle ways [15]; this paper shows how to exploit that dependency.”
Datamodels: Predicting Predictions from Training Data (2022, background): “A recent body of work in machine learning suggests that the answers to these questions lie within both the learning algorithm and the training data used [GDG17; CLK+19; IST+19; Hoo21; JTM21].”
A Systematic Study of Bias Amplification (2022, background): “models beyond biases in the training data [21].”
Benchmarking Bias Mitigation Algorithms in Representation Learning through Fairness Metrics (2021, background): “There is increasing literature studying how biased datasets can bias learning algorithms to discriminate [1, 2, 6, 7, 11, 12, 13].”
Algorithm fairness in artificial intelligence for medicine and healthcare (2023, background): “However, despite not using race as a covariate, a retrospective study found that COMPAS assigned medium-to-high risk scores to Black defendants twice as often than to White defendants (45% versus 23%, respectively) 4 .”
Large language models encode clinical knowledge (2022, background): “algorithmic design choices [32], and differences in behavior or performance of machine learning systems across populations and groups that introduce downstream harms when used to inform medical decision making [13].”