Documenting the English Colossal Clean Crawled Corpus

By Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, Matt Gardner. 2021

51 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

Citations in the corpus

Challenges in Detoxifying Language Models (2021, positive): “We note that AAE tweets already have substantially higher loss under a standard LM (5.53 vs. 4.77), which is likely a result of the underrepresentation (0.07% of all documents) of AAE in C4, as highlighted by Dodge et al. (2021).”
Mitigating harm in language models with conditional-likelihood filtration (2021, methodology): “It also erroneously ﬂags non-harmful webpages which use blocklisted words in academic, rhetorical, or expository contexts, and has been shown to disproportionately ﬁlter out text associated with minority identities [Dodge et al., 2021].”
MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, methodology): “[32] Jesse Dodge, Maarten Sap, Ana Marasovic, William Agnew, Gabriel Ilharco, Dirk Groeneveld, and Matt Gardner.”
Multimodal datasets: misogyny, pornography, and malignant stereotypes (2021, background): “But without careful contextual analysis, ﬁltering mechanisms are likely to censor and erase marginalized experiences [71].”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, methodology): “, 2011), LM systems, or bots/templates, as for example in the analysis of C4 (Dodge et al., 2021).”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “There has also been an increased focus on analyzing other dimensions of data quality and stewardship [102, 107, 121, 123], with several noteworthy initiatives aiming to document both existing [9, 20, 30, 43], and newly developed [16, 60, 136] resources.”
Counterfactual Memorization in Neural Language Models (2021, neutral): “Our observation of near-duplicated training-validation document pairs is consistent with recent studies that identifies data contamination in large Internet crawled text corpus [Lee et al., 2021, Dodge et al., 2021].”
What Does it Mean for a Language Model to Preserve Privacy? (2022, background): “In Web-scraped datasets [76, 78] that are commonly used in training large language models [22] it is typically not possible to unambiguously map individual pieces of information to speciﬁc “owners”.”
Measuring Forgetting of Memorized Training Examples (2022, methodology): “We train decoder-only, 110M parameter, language models (with the T5 codebase and architecture from [64]) for one epoch over a version of C4 ([19]).”
Whose Language Counts as High Quality? Measuring Language Ideologies in Text Data Selection (2022, background): “Bad-word ﬁlter-ing, which removes documents that contain certain stop-words, disproportionately excludes language about and by minority groups (Dodge et al., 2021).”
Addressing "Documentation Debt" in Machine Learning: A Retrospective Datasheet for BookCorpus (2021, methodology): “For instance, [13] provides retrospective documentation for the Colossal Clean Crawled Corpus (C4, introduced by [39]), and [7] analyzes”
Challenges and Strategies in Cross-Cultural NLP (2022, background): “Relevant to the impact of the source of data, Dodge et al. (2021) discuss how the Colossal Clean Crawled Corpus (C4; Raffel et al., 2020), an English web-based corpus, is skewed in favour of US governmental institutions and main-stream US media, while data ﬁlter-ing, which tries to remove slurs, or…”
SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets (2021, background): “Moreover, with the growth in popularity of pre-trained language models, inadvertent cross-contamination of training and test sets is a real risk [15].”
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2021, background): “Domain 3: News Articles News articles make up a significant part of the data commonly used to train LMs (Dodge et al., 2021).”