Understanding Unequal Gender Classification Accuracy from Face Images

By Vidya Muthukumar, Tejaswini Pedapati, Nalini K. Ratha, Prasanna Sattigeri, C. Wu, Brian Kingsbury, Abhishek Kumar, Samuel Thomas, Aleksandra Mojsilovic, Kush R. Varshney. 2018

53 citations recorded by the source corpus.

Recent work shows unequal performance of commercial face classification services in the gender classification task across intersectional groups defined by skin type and gender. Accuracy on dark-skinned females is significantly worse than on any other group. In this paper, we conduct several analyses to try to uncover the reason for this gap. The main finding, perhaps surprisingly, is that skin type is not the driver. This conclusion is reached via stability experiments that vary an image's skin type via color-theoretic methods, namely luminance mode-shift and optimal transport. A second suspect, hair length, is also shown not to be the driver via experiments on face images cropped to exclude the hair. Finally, using contrastive post-hoc explanation techniques for neural networks, we bring forth evidence suggesting that differences in lip, eye and cheek structure across ethnicity lead to the differences. Further, lip and eye makeup are seen as strong predictors for a female face, which is a troubling propagation of a gender stereotype.

Subfields: Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs (2021, methodology): “[59] used computer vision post-processing techniques and post-hoc explanation methods to uncover potential causes of the performance”
Demographic Effects in Facial Recognition and Their Dependence on Image Acquisition: An Evaluation of Eleven Commercial Systems (2019, background): “Recent work found no effect of skin pixel intensity on gender classiﬁ-cation accuracy [20] suggesting that other factors underlie differences in classiﬁcation performance.”
How Computers See Gender (2019, background): “Differences in true positive accuracy likely indicate how these models are trained to recognize contrasting “female” and “male” features (e.g., lips and cheekbones [76]).”
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging (2023, background): “tigating what is depicted in the most salient region of an image) (Hendricks et al. 2018; Jia, Lansdall-Welfare, and Cristianini 2018; Muthukumar et al. 2018; Singh et al. 2020).”
Towards causal benchmarking of bias in face analysis algorithms (2020, methodology): “The triple correlation between hair length, gender and skin color had been noticed in a previous study [56].”