Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research

By Luca Soldaini, Rodney Kinney, Akshita Bhagia, Dustin Schwenk, David Atkinson, Russell Authur, Ben Bogin, Khyathi Raghavi Chandu, Jennifer Dumas, Yanai Elazar and 26 more. Annual Meeting of the Association for Computational Linguistics, 2024

519 citations recorded by the source corpus.

Information about pretraining corpora used to train the current best-performing language models is seldom discussed: commercial models rarely detail their data, and even open models are often released without accompanying training data or recipes to reproduce them. As a result, it is challenging to conduct and advance scientific research on language modeling, such as understanding how training data impacts model capabilities and limitations. To facilitate scientific research on language model pretraining, we curate and release Dolma, a three-trillion-token English corpus, built from a diverse mixture of web content, scientific papers, code, public-domain books, social media, and encyclopedic materials. We extensively document Dolma, including its design principles, details about its construction, and a summary of its contents. We present analyses and experimental results on intermediate states of Dolma to share what we have learned about important data curation practices. Finally, we open-source our data curation toolkit to enable reproduction of our work as well as support further research in large-scale data curation.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/ad1bb59e3e18a0dd8503c3961d6074f162baf710

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

WildChat: 1M ChatGPT Interaction Logs in the Wild (2024, unknown): “While the base model training data is readily available (Soldaini et al., 2024), the crucial instruction-tuning datasets are often proprietary, leading to a gap in accessibility for researchers who wish to advance the field.”
The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale (2024, methodology): “Finally, Dolma [38] has a Common Crawl-based subset that uses fastText for language classification, heuristic rules from MassiveText and C4 for quality filtering, rules-and classifier-based toxicity filtering, and URL, document and paragraph-level deduplication using a Bloom filter.”
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024, methodology): “However, since we continue to SFT the model after the decay stage, we do not utilize the final checkpoints.”
Dream 7B: Diffusion Large Language Models (2025, unknown): “7 (Soldaini et al., 2024) for general domain text, OpenCoder (Huang et al., 2025) for programming and code datasets, and DCLM-Baseline (Li et al., 2024) for curated high-quality text and mathematical content.”
OLMo: Accelerating the Science of Language Models (2024, methodology): “We built our training dataset out of a 2T-token sample from our open dataset, Dolma (Soldaini et al., 2024), which we describe in Section 2.2.”
mmBERT: A Modern Multilingual Encoder with Annealed Language Learning (2025, unknown): “From Dolma (Soldaini et al., 2024) we use StarCoder, Stackexchange, Arxiv, and PeS2o.”
Seq vs Seq: An Open Suite of Paired Encoders and Decoders (2025, unknown): “7 (Soldaini et al., 2024).”