CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data

By Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco (Paco) Guzmán, Armand Joulin, Edouard Grave. 2019

830 citations recorded by the source corpus.

Pre-training text representations have led to significant improvements in many areas of natural language processing. The quality of these models benefits greatly from the size of the pretraining corpora as long as its quality is preserved. In this paper, we describe an automatic pipeline to extract massive high-quality monolingual datasets from Common Crawl for a variety of languages. Our pipeline follows the data processing introduced in fastText (Mikolov et al., 2017; Grave et al., 2018), that deduplicates documents and identifies their language. We augment this pipeline with a filtering step to select documents that are close to high quality corpora like Wikipedia.

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Evaluation & Benchmarks

Citations in the corpus

LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We preprocess five CommonCrawl dumps, ranging from 2017 to 2020, with the CCNet pipeline (Wenzek et al., 2020).”
Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “We also use documents from the 2020-10 common crawl dump, preprocessed with the CCNet pipeline (Wenzek et al., 2020).”
Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “In our experiments, we mainly compare GPT-J and the following models: • GPT-J + CC : GPT-J finetuned on C , our subset of CCNet without any API calls.”
How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, background): “This behavior is potentially useful because 1) the knowledge is built up by pre-training on unstructured and unlabeled text data, which is freely available in huge quantities on the Internet (Raffel et al., 2019; Wenzek et al., 2019), and 2) it is possible to retrieve information using informal natu”
Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “Datasets We pre-train on 25 languages (CC25) extracted from the CC corpora (Wenzek et al., 2019; Conneau et al., 2019).”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, neutral): “…CC-Stories (Trinh and Le, 2018) contains a subset of CommonCrawl data filtered to match the story-like style of Winograd schemas (31GB); • English CC100 (Wenzek et al., 2020), a dataset extracted from CommonCrawl snapshots between January 2018 and December 2018, filtered to match the style…”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “, 2020), CCNET (Wenzek et al., 2020), and MC4 (Xue et al.”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “Many pre-trained versions of XLM have been released; the most massively-multilingual variant was trained on 100 languages from Wikipedia.”
Beyond English-Centric Multilingual Machine Translation (2020, methodology): “CCMatrix works on the large monolingual corpora in the 91 languages of CCNet (Wenzek et al., 2019), but at this scale, the global search is computationally demanding even with fast indexing from FAISS (Johnson et al., 2019).”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “We first filter URLs to remove adult content using a blocklist and a score sys-tem (Appendix G.1), we then use trafilatura (Barbaresi, 2021) to extract content from pages, and perform language identification with the fastText classifier from CCNet (Wenzek et al., 2020).”
Re2G: Retrieve, Rerank, Generate (2022, methodology): “S PHERE is built from CCNet [Wenzek et al., 2020] and over twenty times the size of the Wikipedia corpus.”
Improving Wikipedia verifiability with AI (2022, positive): “Both the Wikipedia snapshot we consider ( i.e. , from KILT (Petroni et al., 2021)) as well as the web snapshot ( i.e. , a CCNet (Wenzek et al., 2019) dump from Sphere (Piktus et al., 2021) which contains 134M web articles, split into 906.3M passages) are from August 2019.”
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, methodology): “Using the Common Crawl dump (Wenzek et al., 2020), a given input example is a single web document, which we randomly cut at an arbitrary point, and only keep the beginning (in order to model left to right generation).”
The Web Is Your Oyster - Knowledge-Intensive NLP against a Very Large Web Corpus (2021, methodology): “This surprisingly high number can be partly explained by how S PHERE was constructed - the head CCNet tier we used contains the documents with the lowest perplexity under a Wikipedia-based language model, favoring the inclusion of wiki-based passages into the corpus.”
Unsupervised Dense Information Retrieval with Contrastive Learning (2021, methodology): “We also performed extensive ablation experiments, and observed that independent random cropping seems to be a strong alternative to the inverse Cloze task for training retrievers. data (Wenzek et al., 2020), where half the batches are sampled from each source.”
PEER: A Collaborative Language Model (2022, methodology): “For each document identifier, we get the corresponding document from CCNet (Wenzek et al., 2020).”
Self-training Improves Pre-training for Natural Language Understanding (2020, methodology): “As a large-scale external bank of unannotated sentences, we extract and ﬁlter text from Common-Crawl 1 (Wenzek et al., 2019).”
Cross-lingual Retrieval for Iterative Self-Supervised Training (2020, background): “We refer the reader to [30, 36] for a detailed description of how the monolingual data are preprocessed.”
LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (2022, neutral): “These transformers have been pretrained on Books (Zhu et al., 2015), English Wikipedia, CC-News (Nagel, 2016), OpenWebText (Gokaslan and Cohen, 2019), CC-Stories (Trinh and Le, 2018), and English CC100 (Wenzek et al., 2020).”