Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus

By Jesse Dodge, Ana Marasovic, Gabriel Ilharco, Dirk Groeneveld, Margaret Mitchell, Matt Gardner, William Agnew. 2021

696 citations recorded by the source corpus.

Large language models have led to remarkable progress on many NLP tasks, and researchers are turning to ever-larger text corpora to train them. Some of the largest corpora available are made by scraping significant portions of the internet, and are frequently introduced with only minimal documentation. In this work we provide some of the first documentation for the Colossal Clean Crawled Corpus (C4; Raffel et al., 2020), a dataset created by applying a set of filters to a single snapshot of Common Crawl. We begin by investigating where the data came from, and find a significant amount of text from unexpected sources like patents and US military websites. Then we explore the content of the text itself, and find machine-generated text (e.g., from machine translation systems) and evaluation examples from other benchmark NLP datasets. To understand the impact of the filters applied to create this dataset, we evaluate the text that was removed, and show that blocklist filtering disproportionately removes text from and about minority individuals. Finally, we conclude with some recommendations for how to created and document web-scale datasets from a scrape of the internet.

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Evaluation & Benchmarks

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “Removing these associations, however, is non-trivial; for instance, filtering off content that is deemed toxic by an automated tool may disproportionately exclude content about or authored by marginalized subgroups in the training data (Dodge et al., 2021).”
Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “, 2021b), but empirical work to validate this claim is still ongoing (Dodge et al., 2021; Smith and Williams, 2021; Tal et al., 2022), and further evidence from up-to-date models is still needed.”
Few-shot Learning with Retrieval Augmented Language Models (2022, background): “6%), likely because it was pre-trained on data pre-dating 2020 (Dodge et al., 2021).”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “…data maintenance practices, to Bender et al. (2021) for a survey issues of using large web-scraped datasets, and to Luccioni and Viviano (2021) and Dodge et al. (2021) for analyses of such large-scale web-scraped corpora and their representational, legal, consent, and PII issues. text with AAVE…”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “In one case, the use of a block list to remove “pornographic” text was shown to also suppress LGBTQ+ and African American English (AAE) text from a corpus (Dodge et al., 2021).”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “Work examining and criticizing datasets (Kreutzer et al., 2022; Dodge et al., 2021; Birhane et al., 2021), functionality (Smart, 2021; Zhang et al., 2021; Carlini et al., 2022; Biderman and Raff, 2022), evaluation and deployment procedures (Biderman and Scheirer, 2020; Talat et al., 111 2022), and…”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “This can overly impact minorities (Dodge et al., 2021), motivating the adoption of practices such as pseudo-crawling, wherein allowed URLs are manually curated (Laurenc¸on et al., 2022).”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “ples; additionally, removing data from different dialects like AAE), unintentionally exacerbating risks of marginalization and exclusion (Dodge et al., 2021)).”
Re-contextualizing Fairness in NLP: The Case of India (2022, background): “Further, the risk of unintentionally excluding marginalized communities based on dialect or other linguistic features while filtering data to ensure quality (Dodge et al., 2021; Gururangan et al., 2022) is even higher in the Indian context because of very limited computational representation of marg”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “T5 (Raffel et al., 2020) The original paper did not release its training data, but it did release code for producing it which was subsequently run and released by Dodge et al. (2021).”
LongForm: Effective Instruction Tuning with Reverse Instructions (2023, positive): “…create three LongForm models by finetuning language models on the LongForm-C dataset to assess their effectiveness on instruction-following text generation: LongForm-T5-3B , based on T5-3B (Raffel et al., 2020; Lester et al., 2021), an encoder-decoder LM, and two autoregressive LMs: LongForm-OPT-6.”
Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “…different from the training data of LLMs, which consists of language found in what we call static texts: documents that are written for a general audience and are relatively self-contained and topically focused (e.g., news articles, books, Wikipedia articles; Gao et al., 2020; Dodge et al., 2021).”
Data Portraits: Recording Foundation Model Training Data (2023, methodology): “Other implementations might include a full-text search using commercial software or a conventional database (Piktus, 2022; Dodge et al., 2021).”
The ROOTS Search Tool: Data Transparency for LLMs (2023, background): “datasets renders them hard to properly curate: we now know that the data used for training LLMs may contain synthetic data (Dodge et al., 2021), privacy-infringing data (Carlini et al.”
BigScience: A Case Study in the Social Construction of a Multilingual Large Language Model (2022, background): “…and to how prioritizing efﬁciency and technical performance comes at the expense of social considerations for datasets (Scheuerman et al., 2021; Birhane et al., 2021b); including over-relying on automatic curation that fails to examine the additional biases it introduces (Dodge et al., 2021).”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, methodology): “Other LM pre-training datasets that have been documented and analyzed include C4 (Dodge et al., 2021; Luccioni and Viviano, 2021; Kreutzer et al., 2022), OSCAR (Kreutzer et al.”
PaLM 2 Technical Report (2023, background): “Documenting pre-training data is a challenging but critical element of understanding language models (Paullada et al., 2021; Dodge et al., 2021; Luccioni & Viviano, 2021).”