A Systematic Study of Bias Amplification

By Melissa Hall, L. V. D. van der Maaten, Laura Gustafson, Aaron B. Adcock. 2022

103 citations recorded by the source corpus.

Recent research suggests that predictions made by machine-learning models can amplify biases present in the training data. When a model amplifies bias, it makes certain predictions at a higher rate for some groups than expected based on training-data statistics. Mitigating such bias amplification requires a deep understanding of the mechanics in modern machine learning that give rise to that amplification. We perform the first systematic, controlled study into when and how bias amplification occurs. To enable this study, we design a simple image-classification problem in which we can tightly control (synthetic) biases. Our study of this problem reveals that the strength of bias amplification is correlated to measures such as model accuracy, model capacity, model overconfidence, and amount of training data. We also find that bias amplification can vary greatly during training. Finally, we find that bias amplification may depend on the difficulty of the classification task relative to the difficulty of recognizing group membership: bias amplification appears to occur primarily when it is easier to recognize group membership than class membership. Our results suggest best practices for training machine-learning models that we hope will help pave the way for the development of better mitigation strategies. Code can be found at https://github.com/facebookresearch/cv_bias_amplification.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “This is particularly concerning as one of the best established phenomena in the study of bias in deep learning models is bias amplification—the fact that social biases in deep learning models tend to be more extreme than those found in their training data (Zhao et al., 2017; Hirota et al., 2022; Hal”
The Platonic Representation Hypothesis (2024, background): “It has been shown that large models can exacerbate existing biases present in their training data (Hall et al., 2022).”
Data Feedback Loops: Model-driven Amplification of Dataset Biases (2022, background): “Machine learning models have a tendency to amplify at test-time biases that exist in their training data, a problem known as bias ampliﬁcation [12, 38, 22].”