Multimodal datasets: misogyny, pornography, and malignant stereotypes

By Abeba Birhane, Vinay Uday Prabhu, Emmanuel Kahembwe. 2021

486 citations recorded by the source corpus.

We have now entered the era of trillion parameter machine learning models trained on billion-sized datasets scraped from the internet. The rise of these gargantuan datasets has given rise to formidable bodies of critical work that has called for caution while generating these large datasets. These address concerns surrounding the dubious curation practices used to generate these datasets, the sordid quality of alt-text data available on the world wide web, the problematic content of the CommonCrawl dataset often used as a source for training large language models, and the entrenched biases in large-scale visio-linguistic models (such as OpenAI's CLIP model) trained on opaque datasets (WebImageText). In the backdrop of these specific calls of caution, we examine the recently released LAION-400M dataset, which is a CLIP-filtered dataset of Image-Alt-text pairs parsed from the Common-Crawl dataset. We found that the dataset contains, troublesome and explicit images and text pairs of rape, pornography, malign stereotypes, racist and ethnic slurs, and other extremely problematic content. We outline numerous implications, concerns and downstream harms regarding the current state of large scale datasets while raising open questions for various stakeholders including the AI community, regulators, policy makers and data subjects.

Subfields: Clinical Psychology, Computer Vision and Pattern Recognition, Gender Studies, Safety, Robustness & Interpretability, AI Ethics, Fairness & Society

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Clinical Psychology

In Computer Vision and Pattern Recognition

In Gender Studies

In Safety, Robustness & Interpretability

In AI Ethics, Fairness & Society

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “In yet another project, a filtering approach that relied on a machine learning image-text alignment model was shown to exacerbate its biases in the created multimodal dataset (Birhane et al., 2021).”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “We also note that these benefits are not hypothetical, as a number of papers about the limits and ethics of LLMs has been explicitly enabled by the public release of previous models (Zhang et al., 2021; Kandpal et al., 2022; Carlini et al., 2022; Birhane et al., 2021; nostalgebraist, 2020; Meng et a”
CM3: A Causal Masked Multimodal Model of the Internet (2022, background): “Birhane et al. (2021) gave in-depth criticisms of Common Crawl based multi-modal datasets and showed the existence of highly problematic examples (i.e., explicit images and text pairs of rape, pornography, and ethnic slurs).”
SLIP: Self-supervision meets Language-Image Pre-training (2021, background): “CLIP’s ability to leverage noisy and minimally ﬁltered data scraped from the open internet has already spurred researchers to begin collecting data in a more care-less manner than previously possible for supervised learning [3].”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “[11] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe.”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, background): “After the release of LAION-400M, several groups (e.g., [6]) already used such tools and investigated potential problems arising from an unfiltered dataset.”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, background): “[4] Abeba Birhane, Vinay Uday Prabhu, and Emmanuel Kahembwe.”
Measuring Data (2022, background): “Analysing multimodal datasets, such as the recent large dataset LAION (Schuhmann et al., 2021), presents unique measurement challenges that we also do not address: Different modalities of a dataset can interact to result in novel harms and biases, as illustrated by Birhane et al. (2021).”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “There has also been an increased focus on analyzing other dimensions of data quality and stewardship [102, 107, 121, 123], with several noteworthy initiatives aiming to document both existing [9, 20, 30, 43], and newly developed [16, 60, 136] resources.”
Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources (2022, background): “…Gehrmann et al., 2021; Wang et al., 2021) and even to retrospectively document and analyze datasets that were developed and released by others without thorough documentation (Bandy and Vincent, 2021; Kreutzer et al., 2021; Birhane et al., 2021; Dodge et al., 2021).”
Grounding Language Models to Images for Multimodal Generation (2023, background): “However, for deployment of such technologies (and future research on generative multi-modal dialogue models), it is essential to test and analyze data used to mitigate the risk of training large multimodal models (Birhane et al., 2021).”
Stable Bias: Evaluating Societal Representations in Diffusion Models (2023, unknown): “…are many sources from which biases can originate in TTI systems; for instance, the training data used for training diffusion such as Stable Diffusion is scraped from the Web and has been shown to contain harmful and pornographic content [11, 37] as well as mislabeled and corrupted examples [59].”
Participation Is not a Design Fix for Machine Learning (2020, background): “labels [5], that they were collected without adequately informed consent [17], and that they are imbalanced across demographic groups in ways that lead to bias in ML systems trained on such datasets [69].”
The MiniPile Challenge for Data-Efficient Language Models (2023, background): “Potential issues with such include token repetitions [33, 21, 1], misogyny, pornography, and malignant stereotypes [9, 7], benchmark data contamination [11, 14], spurious correlations [27, 37, 38], diluted robustness due to data mixing [45] and potentially sensitive information [20].”
The political ideology of conversational AI: Converging evidence on ChatGPT's pro-environmental, left-libertarian orientation (2023, background): “Proceedings of the National Academy of Sciences, 118(8): e2017548118, 2021.”
DALL-EVAL: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models (2022, background): “Recent work has reported social biases in vision-and-language datasets and models learned from them [50, 6].”