Characterizing the Variability in Face Recognition Accuracy Relative to Race

By S. KrishnapriyaK., Kushal Vangara, Michael C. King, Vítor Albiero, Kevin W. Bowyer. 2019

98 citations recorded by the source corpus.

Many recent news headlines have labeled face recognition technology as "biased" or "racist". We report on a methodical investigation into differences in face recognition accuracy between African-American and Caucasian image cohorts of the MORPH dataset. We find that, for all four matchers considered, the impostor and the genuine distributions are statistically significantly different between cohorts. For a fixed decision threshold, the African-American image cohort has a higher false match rate and a lower false non-match rate. ROC curves compare verification rates at the same false match rate, but the different cohorts achieve the same false match rate at different thresholds. This means that ROC comparisons are not relevant to operational scenarios that use a fixed decision threshold. We show that, for the ResNet matcher, the two cohorts have approximately equal separation of impostor and genuine distributions. Using ICAO compliance as a standard of image quality, we find that the initial image cohorts have unequal rates of good quality images. The ICAO-compliant subsets of the original image cohorts show improved accuracy, with the main effect being to reducing the low-similarity tail of the genuine distributions.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, AI Ethics, Fairness & Society, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In AI Ethics, Fairness & Society

In Signal Processing

Citations in the corpus

One Label, One Billion Faces: Usage and Consistency of Racial Categories in Computer Vision (2021, methodology): “Audits of algorithms, APIs, and software [39],[17],[27] have found statistically significant disparities in performance across groups and have been critical to bringing racial disparities to light.”
Towards causal benchmarking of bias in face analysis algorithms (2020, methodology): “ like error frequencies on a test set that is sampled in the wild, hopefully mirroring some of the data statistics that will be encountered in the ﬁeld. Studies of algorithmic bias in computer vision [12, 10, 43, 47] have adapted this approach by adding one additional step: each image of the test se”