FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age for Bias Measurement and Mitigation
By Kimmo Kärkkäinen, Jungseock Joo. 2021
785 citations recorded by the source corpus.
Existing public face image datasets are strongly biased toward Caucasian faces, and other races (e.g., Latino) are significantly underrepresented. The models trained from such datasets suffer from inconsistent classification accuracy, which limits the applicability of face analytic systems to non-White race groups. To mitigate the race bias problem in these datasets, we constructed a novel face image dataset containing 108,501 images which is balanced on race. We define 7 race groups: White, Black, Indian, East Asian, Southeast Asian, Middle Eastern, and Latino. Images were collected from the YFCC-100M Flickr dataset and labeled with race, gender, and age groups. Evaluations were performed on existing face attribute datasets as well as novel image datasets to measure the generalization performance. We find that the model trained from our dataset is substantially more accurate on novel datasets and the accuracy is consistent across race and gender groups. We also compare several commercial computer vision APIs and report their balanced accuracy across gender, race, and age groups. Our code, data, and models are available at https://github.com/joojs/fairface.
Subfields: Computer Vision and Pattern Recognition, AI Ethics, Fairness & Society, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Ming-Hsuan Yang
- Andrew Zisserman
- Cordelia Schmid
In AI Ethics, Fairness & Society
- Ali Farhadi
- James Y. Zou
- John V. Guttag
- Douwe Kiela
- Harini Suresh
- Matthieu Cord
- Ryan Shaun Joazeiro de Baker
- Aaron Hawn
In Signal Processing
- Chen Change Loy
- Alexander Kolesnikov
- Arsha Nagrani
- Eli Shechtman
- Wayne Wu
- Pang Wei Koh
- Xiaohua Zhai
- A. Dimakis
Citations in the corpus
- A Step Toward More Inclusive People Annotations for Fairness (2021, background): “, Celeb-A [18], FairFace [14], UTKFace [32]) but may also be useful for studying bias [1, 23].”
- CelebV-HQ: A Large-Scale Video Facial Attributes Dataset (2022, methodology): “, attribute recognition [97,17,41], action recognition [81,37], emotion recognition [18,47] and forgery detection [50,27,104]).”
- StyleSDF: High-Resolution 3D-Consistent Image and Geometry Generation (2021, background): “A possible solution is creating a more balanced dataset, e.g., as in [35].”
- DataComp: In search of the next generation of multimodal datasets (2023, methodology): “Furthermore, we find that while the BYOD model improves accuracy, in FairFace most of this improvement is on men (ranging from 1.7pp gain to 9.9pp gain), while on women, it offers little change (ranging from 0.6pp gain to 6.2pp drop).”
- A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle (2019, background): “In other cases, it may be better to develop entirely new benchmarks that are more representative and better suited to the task at hand [2, 14, 27].”
- PaLI-X: On Scaling up a Multilingual Vision and Language Model (2023, methodology): “We use the FairFace dataset [85] that comprises of images of people with ground-truth attributes: gender presentation, age and ethnicity.”
- OBELISC: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents (2023, methodology): “We studied the generations and analyzed the trends for each protected characteristic using FairFace (Kärkkäinen and Joo, 2021) and StableBias (Luccioni et al.”
- Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning (2022, methodology): “We performed zero-shot evaluations on CLIP (ViTB/32) on the evaluation set of the FairFace dataset [Kärkkäinen and Joo, 2021], which has 10,954 images.”
- Fake it till you make it: face analysis in the wild using synthetic data alone (2021, background): “This is especially difficult in the realm of human-related computer vision, where concerns about the fairness of models and the ethics of deployment are paramount [31].”
- General Facial Representation Learning in a Visual-Linguistic Manner (2021, methodology): “We compare FaRL with the 400M-data-pretrained CLIP [73]2, as well as the baseline model associated with FairFace [46].”