The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset

By Hugo Laurenccon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, E. G. Ponferrada, Huu Nguyen and 43 more. 2023

217 citations recorded by the source corpus.

As language models grow ever larger, the need for large-scale high-quality text datasets has never been more pressing, especially in multilingual settings. The BigScience workshop, a 1-year international and multidisciplinary initiative, was formed with the goal of researching and training large language models as a values-driven undertaking, putting issues of ethics, harm, and governance in the foreground. This paper documents the data creation and curation efforts undertaken by BigScience to assemble the Responsible Open-science Open-collaboration Text Sources (ROOTS) corpus, a 1.6TB dataset spanning 59 languages that was used to train the 176-billion-parameter BigScience Large Open-science Open-access Multilingual (BLOOM) language model. We further release a large initial subset of the corpus and analyses thereof, and hope to empower large-scale monolingual and multilingual modeling projects with both the data and the processing tools, as well as stimulate research around this large multilingual corpus.

Subfields: Artificial Intelligence, Molecular Biology, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Large Language Models

Citations in the corpus

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, background): “This can overly impact minorities (Dodge et al., 2021), motivating the adoption of practices such as pseudo-crawling, wherein allowed URLs are manually curated (Laurenc¸on et al., 2022).”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “In particular, ROOTS has not been used to train models beyond BLOOM.”
LAION-5B: An open large-scale dataset for training next generation image-text models (2022, neutral): “Community efforts like the The Pile [18] and BigScience ROOTS [40] made large text datasets more accessible.”
Data Portraits: Recording Foundation Model Training Data (2023, methodology): “GPT-3 and other LLMs ([25, 23] among others) use the MinHash sketch of Broder [6] to deduplicate their training sets.”
The ROOTS Search Tool: Data Transparency for LLMs (2023, methodology): “The ROOTS corpus (Laurençon et al., 2022) is a high-quality, heterogeneous, multilingual text corpus collected as part of the BigScience project to train the BLOOM LLM (Scao et al., 2022).”
The Stack: 3 TB of permissively licensed source code (2022, background): “AlphaCode (Li et al., 2022), BLOOM (Laurençon et al., 2022), CodeGen (Nijkamp et al., 2022), and InCoder (Fried et al., 2022) all included this resource in their pretraining dataset.”
Fast-DetectGPT: Efficient Zero-Shot Detection of Machine-Generated Text via Conditional Probability Curvature (2023, unknown): “1 (Scao et al., 2022) bigscience/bloom-7b1 7.1B ROOTS corpus (Laurenc¸on et al., 2022).”
MathPile: A Billion-Token-Scale Pretraining Corpus for Math (2023, unknown): “BLOOM [57] uses the ROOTS dataset [32] for multilingual pre-training.”
Investigating the Translation Performance of a Large Multilingual Language Model: the Case of BLOOM (2023, background): “6TB of text (of which 30% English), from various sources, although 38% of the data, known as the ROOTS corpus (Laurençon et al., 2022),5 is from Oscar web data (Ortiz Suárez et al.”
A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “A lot of efforts have been made to clean raw corpora and create high-quality training corpora for LLMs [124, 295, 145, 296, 204, 268].”
Scaling Data-Constrained Language Models (2023, unknown): “ROOTS filter In addition, we benchmark with the filtering procedure from the ROOTS corpus [51].”