Large image datasets: A pyrrhic win for computer vision?

By Vinay Uday Prabhu, Abeba Birhane. 2020

434 citations recorded by the source corpus.

In this paper we investigate problematic practices and consequences of large scale vision datasets (LSVDs). We examine broad issues such as the question of consent and justice as well as specific concerns such as the inclusion of verifiably pornographic images in datasets. Taking the ImageNet-ILSVRC-2012 dataset as an example, we perform a cross-sectional model-based quantitative census covering factors such as age, gender, NSFW content scoring, class- wise accuracy, human-cardinality-analysis, and the semanticity of the image class information in order to statistically investigate the extent and subtleties of ethical transgressions. We then use the census to help hand-curate a look-up-table of images in the ImageNet-ILSVRC-2012 dataset that fall into the categories of verifiably pornographic: shot in a non-consensual setting (up-skirt), beach voyeuristic, and exposed private parts. We survey the landscape of harm and threats both the society at large and individuals face due to uncritical and ill-considered dataset curation practices. We then propose possible courses of correction and critique their pros and cons. We have duly open-sourced all of the code and the census meta-datasets generated in this endeavor for the computer vision community to build on. By unveiling the severity of the threats, our hope is to motivate the constitution of mandatory Institutional Review Boards (IRB) for large scale dataset curation.

Subfields: Clinical Psychology, Computer Vision and Pattern Recognition, Safety Research, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Clinical Psychology

In Computer Vision and Pattern Recognition

In Safety Research

In Evaluation & Benchmarks

Citations in the corpus

AI and the Everything in the Whole Wide World Benchmark (2021, background): “The consequences of this oversight were made tangible by the work of Birhane and Prabhu [2021] and Crawford and Paglen [2019] which revealed the existence of a range of derogatory and offensive categories in the “person” subtree.”
Understanding and Evaluating Racial Biases in Image Captioning (2021, background): “In line with existing mitigation efforts [76, 58], manual captions containing slurs or other offensive concepts should be removed from the dataset.”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, background): “Recent research [4, 5] described many potential negative societal implications that may arise due to careless use of vision-language models, e.g., the models perform worse for certain groups of users or reproduce discriminatory behavior.”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, background): “Dataset audits have revealed these datasets tend to reflect social stereotypes, oppressive viewpoints, and derogatory, or otherwise harmful, associations to marginalized identity groups [46, 4].”
Multimodal datasets: misogyny, pornography, and malignant stereotypes (2021, background): “Furthermore, such datasets are collected without the consent and awareness of the data subject [11].”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, background): “As the scale of ML research grows, it becomes increasingly difficult to validate automatically collected and curated datasets (Biderman and Scheirer, 2020; Birhane and Prabhu, 2021; Bender et al., 2021).”
Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Development (2021, neutral): “A recent audit of ImageNet found the dataset contained significant gender biases, and even the inclusion of non-consensual pornographic imagery, depicting predominantly women [89].”
Differentially Private Learning Needs Better Features (or Much More Data) (2020, methodology): “5The Tiny Images dataset has been withdrawn after the discovery of offensive class labels (Prabhu & Birhane, 2020).”
The Values Encoded in Machine Learning Research (2021, background): “[36] Vinay Uday Prabhu and Abeba Birhane.”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “[107] Vinay Uday Prabhu and Abeba Birhane.”
A Study of Face Obfuscation in ImageNet (2021, background): “…in the 1000 categories of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) (Russakovsky et al., 2015); nevertheless, the dataset exposes many people co-occurring with other objects in images (Prabhu & Birhane, 2021), e.g., people sitting on chairs, walking dogs, or drinking beer (Fig.”
Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources (2022, background): “Data documentation to support transparency has gained traction following calls for a reevaluation of the treatment of data in machine learning (ML) at large (Prabhu and Birhane, 2020; Jo and Gebru, 2020; Paullada et al., 2021; Gebru et al., 2021).”
Artsheets for Art Datasets (2021, background): “Dataset audits have uncovered a host of concerns with the ways minoritized groups are represented – or fail to be represented at all – in prominent ML datasets [17, 18, 19, 20].”
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications (2022, background): “Datasets that are not carefully evaluated may not be well-matched to what they aim to measure (Blodgett et al., 2021) or give rise to harms (e.g., Prabhu and Birhane, 2021).”
The Forgotten Margins of AI Ethics (2022, background): “Many of these papers also did not address questions such as how the dataset is sourced and curated, what was used as input data to build the algorithm, how historical and current stereotypes are encoded [7, 9], and whether improving recidivism algorithms results in a net harm to marginalized groups.”