C4Corpus: Multilingual Web-size Corpus with Free License

By Ivan Habernal, Omnia Zayed, Iryna Gurevych. 2016

42 citations recorded by the source corpus.

Large Web corpora containing full documents with permissive licenses are crucial for many NLP tasks. In this article we present the construction of 12 million-pages Web corpus (over 10 billion tokens) licensed under CreativeCommons license family in 50+ languages that has been extracted from CommonCrawl, the largest publicly available general Web crawl to date with about 2 billion crawled URLs. Our highly-scalable Hadoop-based framework is able to process the full CommonCrawl corpus on 2000+ CPU cluster on the Amazon Elastic Map/Reduce infrastructure. The processing pipeline includes license identification, state-of-the-art boilerplate removal, exact duplicate and near-duplicate document removal, and language detection. The construction of the corpus is highly configurable and fully reproducible, and we provide both the framework (DKPro C4CorpusTools) and the resulting data (C4Corpus) to the research community.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “, 2019c), comprise only Creative Commons content (Habernal et al., 2016), or are focused on parallel training data for machine translation (Smith et al.”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, methodology): “Limitations and Recommendations We recognize that we have only examined some of the possible issues with a dataset of this size, and so in addition to making the dataset available to download, we recommend providing a location for others to report issues they find (Habernal et al., 2016; Schäfer, 20”
The Argument Reasoning Comprehension Task: Identification and Reconstruction of Implicit Warrants (2017, methodology): “As another baseline, we used a 4-gram Modified Kneser-Ney language model trained on 500M tokens (100k vocabulary) from the C4Corpus (Habernal et al., 2016).”
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, methodology): “It includes text from the RedPajama dataset [32], C4 [68], Wikipedia, and stack overﬂow.”
Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction (2021, background): “Corpora based on machine-checked licenses have also been developed (Habernal et al., 2016), as well as systems to merge annotation with web parts from the CommonCrawl (Sch¨afer, 2016).”