Diversity in Faces

By Michele Merler, Nalini K. Ratha, Rogério Schmidt Feris, John R. Smith. 2019

217 citations recorded by the source corpus.

Face recognition is a long standing challenge in the field of Artificial Intelligence (AI). The goal is to create systems that accurately detect, recognize, verify, and understand human faces. There are significant technical hurdles in making these systems accurate, particularly in unconstrained settings due to confounding factors related to pose, resolution, illumination, occlusion, and viewpoint. However, with recent advances in neural networks, face recognition has achieved unprecedented accuracy, largely built on data-driven deep learning methods. While this is encouraging, a critical aspect that is limiting facial recognition accuracy and fairness is inherent facial diversity. Every face is different. Every face reflects something unique about us. Aspects of our heritage - including race, ethnicity, culture, geography - and our individual identify - age, gender, and other visible manifestations of self-expression, are reflected in our faces. We expect face recognition to work equally accurately for every face. Face recognition needs to be fair. As we rely on data-driven methods to create face recognition technology, we need to ensure necessary balance and coverage in training data. However, there are still scientific questions about how to represent and extract pertinent facial features and quantitatively measure facial diversity. Towards this goal, Diversity in Faces (DiF) provides a data set of one million annotated human face images for advancing the study of facial diversity. The annotations are generated using ten well-established facial coding schemes from the scientific literature. The facial coding schemes provide human-interpretable quantitative measures of facial features. We believe that by making the extracted coding schemes available on a large set of faces, we can accelerate research and development towards creating more fair and accurate facial recognition systems.

Subfields: Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

AI and the Everything in the Whole Wide World Benchmark (2021, background): “…and gender-nonconforming gender identities and non-white, non-Western racial identities are underrepresented in mainstream face datasets [Merler et al., 2019], and images of members of these communities are often tagged with racial or ethnic slurs, even on large general use datasets…”
Saving Face: Investigating the Ethical Concerns of Facial Recognition Auditing (2020, methodology): “For example, IBM’s Diversity in Faces dataset was sourced from Creative Common licensed images uploaded to Flickr [32].”
FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age (2019, methodology): “To date, numerous large scale face image datasets [22, 31, 13, 69, 37, 24, 41, 68, 15, 27, 46, 8, 39] have been proposed and fostered research and development for automated face detection [35, 21], alignment [66, 44], recognition [57, 49], generation [67, 5, 26, 58], modiﬁcation [3, 32, 19], and…”
A Survey on Bias and Fairness in Machine Learning (2019, background): “Diversity in faces dataset [87] 1 million images Facial images”
Differential Privacy Has Disparate Impact on Model Accuracy (2019, methodology): “ Figure 2: Sentiment analysis of tweets and species classiﬁcation. 4.1Gender and age classiﬁcation on facial images Dataset. We use the recently released Flickr-based Diversity in Faces (DiF) dataset [27] and the UTKFace dataset [39] as another source of darker-skinned faces. We use the attached met”
Closing the AI accountability gap: defining an end-to-end framework for internal algorithmic auditing (2020, methodology): “From the datasheet for the CelebA dataset, we see that although the provided binary gender labels seem balanced for this dataset (58.1% female, 42% male), other demographic details are quite skewed (77.8% aged 0-45, 22.1% aged over 46 and 14.2% lighter-skinned, 85.8% darker-skinned)[47].”
On the genealogy of machine learning datasets: A critical history of ImageNet (2021, background): “…proliferation of algorithmic fairness interventions have emerged in recent years that hinge on balancing representation of different demographic groups within training data-sets — the data used for algorithms to “ learn ” associations (Shankar et al., 2017; Merler et al., 2019; Yang et al., 2020).”
A Survey of Race, Racism, and Anti-Racism in NLP (2021, methodology): “the the publicly available YFCC-100M data set and annotated with “craniofacial distances, areas and ratios, facial symmetry and contrast, skin color, age and gender predictions” (Merler et al., 2019).”
FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation (2021, methodology): “A recent work also used the same dataset to construct a huge unfiltered face dataset (Diversity in Faces, DiF) [40].”
Evaluating Deep Neural Networks Trained on Clinical Images in Dermatology with the Fitzpatrick 17k Dataset (2021, methodology): “…the ITA of each full and YCbCr masked image by converting the input image to CIE − LAB color space, which contains L : luminance and B : yellow, and applying the following formula [38]: where L ∗ and B ∗ are the mean of non-masked pixels with values within one standard deviation of the actual mean.”
How We've Taught Algorithms to See Identity: Constructing Race and Gender in Image Databases for Facial Analysis (2020, background): “A number of image databases were created to improve the diversity of available faces for training and evaluation, and thus, ideally, mitigate potential representation biases within facial analysis models (e.g., DiF [109], PPB [24]).”
One Label, One Billion Faces: Usage and Consistency of Racial Categories in Computer Vision (2021, methodology): “Other datasets, such as PPB [7] and Diversity in Faces [32], use phenotype annotations in lieu of racial categories.”
Auditing ImageNet: Towards a Model-driven Framework for Annotating Demographic Attributes of Large-Scale Image Datasets (2019, background): “As Merler et al. note in Diversity in Faces , “Ultimately, it will require an iterative process of understanding diversity to make more balanced data sets and create more fair models” [16].”
Are We Learning Yet? A Meta Review of Evaluation Failures Across Machine Learning (2021, background): “For example, facial recognition benchmarks drastically under-represent darker and female faces [96], making it difﬁcult to perceive when models fail to perform acceptably for this subgroup [7, 20].”
Excavating AI: the politics of images in machine learning training sets (2021, unknown): “25 Merler et al. (2019).”
Algorithmic realism: expanding the boundaries of algorithmic thought (2020, background): “…when research exposed that face and gender recognition systems are more accurate on light-skinned men than on dark-skinned women [19], the primary response was to strive for less biased systems [85, 95, 122, 147, 155], in one case by targeting homeless people of color for facial images [112].”
Face Recognition: Too Bias, or Not Too Bias? (2020, background): “De-mographics like gender, ethnicity, race, and age are underrepresented in most public datasets [24, 38].”