Gender Shades: Intersectional Accuracy Disparities in Commercial Gender Classification

By Joy Buolamwini, Timnit Gebru. 2018

5,467 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Safety, Robustness & Interpretability

Citations in the corpus

Learning Transferable Visual Models From Natural Language Supervision (2021, methodology): “In this section, we provide preliminary analysis of some of the biases in CLIP, using bias probes inspired by those outlined in Buolamwini & Gebru (2018) and Kärkkäinen & Joo (2019).”
Model Cards for Model Reporting (2018, background): “s). No new information is inferred or annotated. Quantitative Analyses Caveats and Recommendations • Does not capture race or skin type, which has been reported as a source of disproportionate errors [5]. • Given gender classes are binary (male/not male), which we include as male/female. Further wor”
AI and the Everything in the Whole Wide World Benchmark (2021, methodology): “A more audit-like methodology, exempliﬁed by Buolamwini and Gebru [2018], creates test sets balanced for sensitive categories so as to be able to test for differential performance across those categories.”
Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs (2021, methodology): “The most notable such piece of work is the Gender Shades study, which used a disaggregated evaluation to show that three commercially available gender classifiers, a type of face-characterization system, had higher error rates for women with darker skin tones than for women or people with darker ski”
Re-imagining Algorithmic Fairness in India and Beyond (2021, background): “often used by researchers as a phenotype [46], but was originally developed to categorise UV sensitivity) [73].”
Shortcut learning in deep neural networks (2020, background): “DNNs can accurately recognise faces, but they show high error rates for faces from minority groups [12].”
Toward Gender-Inclusive Coreference Resolution (2019, background): “For instance, if one evaluates on highly biased data, it will be difficult to capture disparities (akin to the over-representation of light skinned men in computer vision datasets [Buolamwini and Gebru 2018]).”
Datasheets for datasets (2018, background): “Of particular concern are recent examples showing that machine learning models can reproduce or amplify unwanted societal biases reflected in training data [4, 5, 9].”
Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, methodology): “A model that is trained on a dataset that is disproportionately skewed towards one particular demographic group may perform poorly for other demographic groups [43].”
Underspecification Presents Challenges for Credibility in Modern Machine Learning (2020, background): “…in the literature on fairness in machine learning, where examples are stratified by socially salient characteristics like skin type and gender (Buolamwini and Gebru, 2018); the ML for healthcare literature (Obermeyer et al., 2019; Oakden-Rayner et al., 2020), where examples are stratified by…”
Saving Face: Investigating the Ethical Concerns of Facial Recognition Auditing (2020, positive): “algorithmic auditing practices. 2 CELEBSET: A TYPICAL FPT BENCHMARK Datasets such as Face Recognition Vendor Tests (FRVT), IARPA Janus Benchmarks (IJB) [38], and the Pilot Parliaments Benchmark (PPB) [6] have been key to identifying classification bias in the specific tasks of face detection, verifi”
Diagnosing Gender Bias in Image Recognition Systems (2020, methodology): “Although more research has been conducted on text than images, prior studies of images have shown similar patterns (Buolamwini and Gebru 2018).”
FairFace: Face Attribute Dataset for Balanced Race, Gender, and Age (2019, methodology): “The Pilot Parliaments Benchmark (PPB) dataset [7] only used profile photographs of government officials taken in well controlled lighting, which makes it non-inthe-wild.”
Exposing and Correcting the Gender Bias in Image Captioning Datasets and Models (2019, background): “…on them which has been analysed by various works like [Stock and Cisse, 2018, Beigman and Klebanov, 2009, Sweeney, 2013, Flores et al., 2016, Buolamwini and Gebru, 2018, Bolukbasi et al., 2016], show how the blatant gender bias in datasets is inherited by the word embeddings, which play a…”
True Few-Shot Learning with Language Models (2021, methodology): “We evaluate the generalization loss of the algorithm chosen by CV (likewise for MDL): where Reliability is important for many practical or commercial applications where worst-case performance is important, such as image recognition [35, 36], dialogue systems [37, 38], and robotics [39, 40].”
Racial disparities in automated speech recognition (2020, background): “Although the total duration and number of words spoken by black and white speakers in our sample were similar, black speakers uttered fewer unique words (5,651) than white speakers (6,280).”