How We've Taught Algorithms to See Identity: Constructing Race and Gender in Image Databases for Facial Analysis

By Morgan Klaus Scheuerman, Kandrea Wade, Caitlin Lustig, Jed R. Brubaker. 2020

209 citations recorded by the source corpus.

Race and gender have long sociopolitical histories of classification in technical infrastructures-from the passport to social media. Facial analysis technologies are particularly pertinent to understanding how identity is operationalized in new technical systems. What facial analysis technologies can do is determined by the data available to train and evaluate them with. In this study, we specifically focus on this data by examining how race and gender are defined and annotated in image databases used for facial analysis. We found that the majority of image databases rarely contain underlying source material for how those identities are defined. Further, when they are annotated with race and gender information, database authors rarely describe the process of annotation. Instead, classifications of race and gender are portrayed as insignificant, indisputable, and apolitical. We discuss the limitations of these approaches given the sociohistorical nature of race and gender. We posit that the lack of critical engagement with this nature renders databases opaque and less trustworthy. We conclude by encouraging database authors to address both the histories of classification inherently embedded into race and gender, as well as their positionality in embedding such classifications.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Experimental and Cognitive Psychology, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Experimental and Cognitive Psychology

In AI Ethics, Fairness & Society

Citations in the corpus

Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development (2021, neutral): “2 Scheuerman et al. raised concerns regarding the use of gender and racial categories in facial analysis datasets, finding that dataset creators tend to present these classifications as natural or immutable characteristics [98].”
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure (2020, background): “ and standards of data collection within ML have themselves been subject to critique, with scholars identifying insufficient documentation and transparency regarding processes of dataset construction [52, 53, 125], as well as problematic consent practices [113]. The lack of accountability to datafie”
Bringing the People Back In: Contesting Benchmark Machine Learning Datasets (2020, background): “…curriculum (e.g. Goodfellow et al. (2016)), the under-specification or data decisions in publications accompanying new datasets (Geiger et al., 2020; Scheuerman et al., 2020), and the routine undervaluing of the work that goes into the construction of datasets (Heinzerling, 2019; Jo & Gebru,…”
On the genealogy of machine learning datasets: A critical history of ImageNet (2021, background): “O ̣ nu ̣ o ̣ ha, 2016; Denton et al., 2020; Scheuerman et al., 2020).”
Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities (2021, neutral): “Though we choose to focus our analysis on queer identity, we note that the observability and measur-ability of race are also important topics (e.g., Scheuerman et al. 2020).”
Between Subjectivity and Imposition (2020, background): “Moreover, the structures, decisions, actors, and frameworks which shape data annotation should be made explicit [39, 72].”
“It’s Complicated”: Negotiating Accessibility and (Mis)Representation in Image Descriptions of Race, Gender, and Disability (2021, background): “These interviews lead us, along with others [10, 11, 95, 97, 120] to strongly caution the use of AI that analyzes humans and human data, and we extend this explicitly to AI which generates image descriptions of people.”
REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets (2020, background): “This metric of gender label inference also brings up a larger question of which situations, if any, gender labels should ever be assigned (Scheuerman et al., 2020; Hamidi et al., 2018).”
A Systematic Study of Bias Amplification (2022, background): “Moreover, the model developer may not always be able to measure the difficulty of recognizing group membership empirically as doing so may involve developing a model that predicts sensitive attributes—something that model developers may want to avoid [24, 27, 39, 43].”
Image Cropping on Twitter: Fairness Metrics, their Limitations, and the Importance of Representation, Design, and Agency (2021, background): “Many critical race scholars also define race in terms of marked or perceived difference, rather than traceable ancestry in itself [42, 44, 65].”
One Label, One Billion Faces: Usage and Consistency of Racial Categories in Computer Vision (2021, methodology): “We draw on the survey of computer vision datasets in [42], who enumerate the diverse and ill-defined array of racial classification schemes and definitions in computer vision datasets.”
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging (2023, background): “2020), such as gender (Keyes 2018; Scheuerman, Paul, and Brubaker 2019; Scheuerman et al. 2020), character traits (Barlas et al.”
Directional Bias Amplification (2021, background): “very few situations in which predicting sensitive attributes makes sense (Scheuerman et al., 2020; Larson, 2017), so we should carefully consider if this is strictly necessary for target applications.”
Artsheets for Art Datasets (2021, methodology): “The manual process of labeling itself, common in ML dataset development pipelines, carries values in both the deﬁnition of categories and in the annotators assigning labels [63, 64].”