Balanced Datasets Are Not Enough: Estimating and Mitigating Gender Bias in Deep Image Representations

By Tianlu Wang, Jieyu Zhao, Mark Yatskar, Kai-Wei Chang, Vicente Ordonez. 2018

474 citations recorded by the source corpus.

In this work, we present a framework to measure and mitigate intrinsic biases with respect to protected variables –such as gender– in visual recognition tasks. We show that trained models significantly amplify the association of target labels with gender beyond what one would expect from biased datasets. Surprisingly, we show that even when datasets are balanced such that each label co-occurs equally with each gender, learned models amplify the association between labels and gender, as much as if data had not been balanced! To mitigate this, we adopt an adversarial approach to remove unwanted features corresponding to protected variables from intermediate representations in a deep neural network – and provide a detailed analysis of its effectiveness. Experiments on two datasets: the COCO dataset (objects), and the imSitu dataset (actions), show reductions in gender bias amplification while maintaining most of the accuracy of the original models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Harms of Gender Exclusivity and Challenges in Non-Binary Representation in Language Technologies (2021, neutral): “These harms are driven by model and dataset biases due to tainted examples, limited features, and sample size disparities (Wang et al., 2019; Barocas et al., 2019; Tan and Celis, 2019), which are consequences of the non-recognition and a lack of understanding of non-binary genders in society (MAP,…”
Understanding and Evaluating Racial Biases in Image Captioning (2021, background): “One important theme is bias amplification [80, 72, 73], or social biases in the data getting amplified in the trained models.”
Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts (2021, background): “Less speciﬁc to the Web data are the unwanted dataset biases [13, 68, 75] that are prone to ampliﬁcation by machine learning models [11, 82].”
Learning with Selective Forgetting (2021, methodology): “This has brought new challenges to the ﬁeld, covering learning from encrypted data [Gilad-Bachrach et al. , 2016], preventing learning of unintended information [Wang et al. , 2019], privacy preserving localization [ Speciale et al. , 2019a; Speciale et al. , 2019b ] , just to name a few.”
Don’t Take the Easy Way Out: Ensemble Based Methods for Avoiding Known Dataset Biases (2019, background): “A popular approach is to use an adversary to remove information about a target feature, often gender or ethnicity, from a model’s internal representations (Edwards and Storkey, 2016; Wang et al., 2018; Kim et al., 2019).”
Demoting Racial Bias in Hate Speech Detection (2020, methodology): “Adversarial training has been used to remove protected features from word embeddings (Xie et al., 2017; Zhang et al., 2018) and intermediate representations for both texts (Elazar and Goldberg, 2018; Zhang et al., 2018) and images (Edwards and Storkey, 2015; Wang et al., 2018).”
Automatically Neutralizing Subjective Bias in Text (2019, background): “For instance, Bordia and Bowman (2019) introduced a regularization term for the language model to penalize the projection of the word embeddings onto that gender subspace, while Wang et al. (2018)”
Measuring Social Biases in Grounded Vision and Language Embeddings (2020, background): “Straightforward applications of this idea are far from trivial, as Wang et al. (2019) show that merely balancing a dataset by a certain attribute is not enough to eliminate bias.”
Learning What Makes a Difference from Counterfactual Examples and Gradient Supervision (2020, critical): “ample. These images serve as the basis of a number of benchmarks for image detection [36], captioning [14], visual question answering [5], etc. They all inherit the biases inherent to the COCO images [2,30,72] which is an increasing cause of concern. A method to improve generalization in this contex”
Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation (2019, background): “Classical literature on mitigating bias generally operates on simpler (often linear) models [11, 50, 28], which are easier to understand and control; only recently have researchers begun looking at mitigating bias in end-to-end trained deep learning models [16, 2, 40, 18, 48, 25, 30, 36, 47, 17].”
Unlearn Dataset Bias in Natural Language Inference by Fitting the Residual (2019, background): “Typical debiasing approaches aim to remove biased features (e.g., gender and image texture) in the learned representation (Zhao et al., 2017; Wang et al., 2019).”
Learning De-biased Representations with Biased Representations (2019, background): “Comparison against HEX .”
Directional Bias Amplification (2021, methodology): “Further mitigation techniques are outside of our scope, but we look to works like Singh et al. (2020); Wang et al. (2019); Agarwal et al. (2020). very few situations in which predicting sensitive attributes makes sense (Scheuerman et al., 2020; Larson, 2017), so we should carefully consider if this…”
DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generative Transformers (2022, background): “Much research work has been done on evaluating common social biases in image-only [52, 60] and text-only [9, 65] models.”
Are My Deep Learning Systems Fair? An Empirical Study of Fixed-Seed Training (2021, methodology): “Debiasing algorithms: To mitigate biases of DL systems, many debiasing algorithms or bias resistant networks have been proposed and evaluated [10, 15, 16, 17, 45, 47, 54, 59, 66, 83, 98, 99, 106].”