What’s in the Box? An Analysis of Undesirable Content in the Common Crawl Corpus

By Alexandra Sasha Luccioni, Joseph D. Viviano. 2021

144 citations recorded by the source corpus.

Whereas much of the success of the current generation of neural language models has been driven by increasingly large training corpora, relatively little research has been dedicated to analyzing these massive sources of textual data. In this exploratory analysis, we delve deeper into the Common Crawl, a colossal web corpus that is extensively used for training language models. We find that it contains a significant amount of undesirable content, including hate speech and sexually explicit content, even after filtering procedures. We discuss the potential impacts of this content on language models and conclude with future research directions and a more mindful approach to corpus collection and analysis.

Subfields: General Health Professions, Safety, Robustness & Interpretability, Philosophy

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In General Health Professions

In Safety, Robustness & Interpretability

In Philosophy

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…those of benchmarking and data maintenance practices, to Bender et al. (2021) for a survey issues of using large web-scraped datasets, and to Luccioni and Viviano (2021) and Dodge et al. (2021) for analyses of such large-scale web-scraped corpora and their representational, legal, consent,…”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “Luccioni and Viviano (2021) provide some exploratory analysis of undesirable content in Common Crawl, wherein they ﬁnd hatespeech and adult content.”
The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, background): “Along with the benefits, however, come new challenges—lack of structure, inconsistent document quality and noisy or harmful content on one hand (Luccioni and Viviano, 2021), increasing infrastructural demands on the other.”
Data Portraits: Recording Foundation Model Training Data (2023, methodology): “Talbot and Osborne [34] use Bloom filters to construct language models while others used similar structures to count features [16, 36].”
The ROOTS Search Tool: Data Transparency for LLMs (2023, background): “crawls contains all kinds of undesired content (Luccioni and Viviano, 2021).”
Multimodal datasets: misogyny, pornography, and malignant stereotypes (2021, background): “More recently, Luccioni and Viviano’s initial study [39] placed the ‘ Hate speech ’ content level to be around 4.02%-5.24% (the 1+ hate n-grams level was estimated higher at 17.78%).”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, background): “Corpora collected by web crawlers are known to be noisy (Junczys-Dowmunt, 2019; Luccioni and Viviano, 2021).”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “[88] Alexandra Sasha Luccioni and Joseph D Viviano.”
Documenting Geographically and Contextually Diverse Data Sources: The BigScience Catalogue of Language Data and Resources (2022, background): “Common Crawl, meanwhile, has been shown to contain hate speech and over-represent sexually explicit content (Luccioni and Viviano, 2021), and typical web-crawling collection practices have no structures for supporting informed consent beyond websites’ own terms and conditions policies that users…”
A Framework for Deprecating Datasets: Standardizing Documentation, Identification, and Communication (2021, background): “However, this focus has started to shift in recent years, during which a flurry of ML scholarship has unearthed many issues with even the most commonplace datasets, showing them to be riddled with errors, perpetuating bias, and intensifying discrimination [30, 42, 57, 68, 75].”
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus (2022, background): “Model-based approaches may help in improving the adult annotation, and should be the next step towards a better annotation of adult content (Luccioni and Viviano, 2021).”
Auditing large language models: a three-layered approach (2023, unknown): “, through ImageNet [22] or CommonCrawl [23]; and the application of these increased compute resources with larger datasets for model pre-training [24].”
MathPile: A Billion-Token-Scale Pretraining Corpus for Math (2023, unknown): “Recent studies have increasingly highlighted the detrimental effects of low-quality and repeated content in pre-training corpora on model training, as evidenced in various works [1, 42, 33, 27, 41].”
LLM Self Defense: By Self Examination, LLMs Know They Are Being Tricked (2023, background): “The very framework that allows LLMs to effectively generate high-quality responses also enables them to generate hateful or otherwise harmful text, as the training corpora are composed of public data containing toxic passages [17].”
BanglaBERT: Language Model Pretraining and Benchmarks for Low-Resource Language Understanding Evaluation in Bangla (2021, background): “Fearing their potentially harmful impacts (Luccioni and Viviano, 2021), we opted not to use them.”
Poisoning Web-Scale Training Datasets is Practical (2023, background): “Many language datasets also contain similarly harmful “hate speech and sexually explicit content, even after filtering” [40].”