Excavating AI: the politics of images in machine learning training sets

By Kate Crawford, Trevor Paglen. 2021

321 citations recorded by the source corpus.

Subfields: Anthropology, Safety Research, Evaluation & Benchmarks, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Anthropology

In Safety Research

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

Citations in the corpus

AI and the Everything in the Whole Wide World Benchmark (2021, methodology): “ImageNet’s scale was a necessary feature to enable the appropriate testing necessary for the introduction of neural nets, however this also disguised problematic dictionary words inherited from WordNet, including several offensive terms and slurs that would later appear as labels in ImageNet [Crawfo”
Diagnosing Gender Bias in Image Recognition Systems (2020, methodology): “As discussed above, GCV is widely used in industry, and unlike its primary competitors, Amazon Rekognition and Microsoft Azure Computer Vision, GCV shares its underlying technology with the world’s largest Internet image search platform (Google Image Search) and other ubiquitous services such as Goo”
Measuring Data (2022, background): “For instance, the well-documented biases of the ImageNet categories (see Crawford and Paglen (2021); Luccioni and Rolnick (2022)) will inﬂuence the results of metrics that rely on these models.”
Multimodal datasets: misogyny, pornography, and malignant stereotypes (2021, background): “This includes stagnant vocabulary of labels [12], misrepresentation of gender [4], prevalence of ethnophaulisms [11] and non-imageability issues in the label space [12, 10, 13].”
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development (2021, neutral): “Previous critical studies of computer vision datasets have been localized in scope to collections of datasets associated with certain tasks or categories (e.g., [25, 89]) or in-depth audits of a handful of commercial models or datasets (e.g., [17, 89]).”
Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructure (2020, background): “, Imagenet was developed in the late 2000s, with certain issues not identified until 2019 [37]).”
On the genealogy of machine learning datasets: A critical history of ImageNet (2021, methodology): “The Archaeology of ImageNet and the genealogical method Crawford and Paglen (2019) begin to examine some of these questions regarding dataset construction in an illumin-ating project, excavating.ai, which they describe as an “ archaeology of datasets.”
Biases in Generative Art: A Causal Look from the Lens of Art History (2020, background): “In a recent art project titled ‘Imagenet Roulette’ [17], AI researcher Kate Crawford and artist Trevor Paglen exposed biases in machine learning datasets.”
Underspecification in Scene Description-to-Depiction Tasks (2022, background): “…is considered noteworthy or unexpected (Van Miltenburg et al., 2016; Misra et al., 2016), in social and cultural stereotyping (van Miltenburg, 2016; Zhao et al., 2017; Otterbacher et al., 2019), and in derogatory and offensive image associations (Birhane et al., 2021; Crawford and Paglen, 2019).”
Taxonomizing and Measuring Representational Harms: A Look at Image Tagging (2023, background): “As another example, researchers have uncovered the presence of lewd tags and images in several widely-used computer vision datasets (Crawford and Paglen 2019; Yang et al. 2020; Prabhu and Birhane 2020).”
Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Research (2021, background): “schemas have been identified in popular image datasets, including poorly-formulated categories and the inclusion of derogatory and offensive labels [8, 9].”
Bugs in the Data: How ImageNet Misrepresents Biodiversity (2022, unknown): “In recent years, following an in-depth study by the Excavating AI project (Crawford and Paglen 2019), an updated official version of the 21K dataset was created, aiming to filter out problematic person categories and to balance their distribution (Yang et al.”