The Pile: An 800GB Dataset of Diverse Text for Language Modeling

By Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima and 2 more. 2020

2,920 citations recorded by the source corpus.

Recent work has demonstrated that increased training dataset diversity improves general cross-domain knowledge and downstream generalization capability for large-scale language models. With this in mind, we present the Pile : an 825 GiB English text corpus tar-geted at training large-scale language models. The Pile is constructed from 22 diverse high-quality subsets—both existing and newly constructed—many of which derive from academic or professional sources. Our evaluation of the untuned performance of GPT-2 and GPT-3 on the Pile shows that these models struggle on many of its components, such as academic writing. Conversely, models trained on the Pile improve signiﬁcantly over both Raw CC and CC-100 on all components of the Pile, while improving performance on downstream evaluations. Through an in-depth exploratory analysis, we document potentially concerning aspects of the data for prospective users. We make publicly available the code used in its construction. 1

Subfields: Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Evaluation & Benchmarks

Citations in the corpus

Training Compute-Optimal Large Language Models (2022, neutral): “For the different evaluation sets in The Pile (Gao et al., 2020), we show the bits-per-byte (bpb) improvement (decrease) of Chinchilla compared to Gopher.”
LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We include two book corpora in our training dataset: the Gutenberg Project, which contains books that are in the public domain, and the Books3 section of ThePile (Gao et al., 2020), a publicly available dataset for training large language models.”
Mixtral of Experts (2024, methodology): “To investigate this, we measure the distribution of selected experts on different subsets of The Pile validation dataset [14].”
Toolformer: Language Models Can Teach Themselves to Use Tools (2023, neutral): “A potential reason for GPT-J not suffering from this problem is that it was trained on more multilingual data than both OPT and GPT-3, including the EuroParl corpus (Koehn, 2005; Gao et al., 2020).”
Unified Scaling Laws for Routed Language Models (2022, methodology): “, 2016], The Pile [Gao et al., 2020], Curation Corpus [Curation, 2020], WikiText-103 [Merity et al.”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, methodology): “Then, following a similar approach as that used to generate Pile-CC in [17], we downloaded and filtered two full CC snapshots (2020-50 and 2021-04).”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “Training ZFLOPs P er pl ex ity 6.0 9.0 12.0 15.0 18.0 21.0 24.0 0.5 1.0 5.0 10.0 Ours (dense) Ours (MoE) (a) In-domain (validation) Training ZFLOPs P er pl ex ity 6.0 9.0 12.0 15.0 18.0 21.0 24.0 0.5 1.0 5.0 10.0 Ours (dense) Ours (MoE) (b) Out-of-domain (the Pile)”
Improving language models by retrieving from trillions of tokens (2021, methodology): “Similar results are found on the Pile dataset (see Fig.”
TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “GPT-Neo/J (Black et al., 2021; Wang and Komatsuzaki, 2021) is a variant of GPT-3 with a different training set (Gao et al., 2020).”
Quantifying Memorization Across Neural Language Models (2022, methodology): “3 These models were trained on a 800 GB dataset that overlaps with The Pile but is not identical and contains data from many new sources, while also removing some data from the Pile.”
Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, background): “Training Data Collection and Duplication Modern language modeling datasets are generated by large-scale scraping of the Internet (Gokaslan et al., 2019; Radford et al., 2019; Raffel et al., 2020; Gao et al., 2020).”
Deduplicating Training Data Makes Language Models Better (2021, background): “Like-wise, more than 5% of the tokens in ~200k sequences outputted by GPT-Neo 1.3B (Black et al., 2021) are part of a 50 token matches of its training data, the Pile (Gao et al., 2020).”
Evaluating Large Language Models Trained on Code (2021, methodology): “Two recent works similar in spirit to Codex are GPT-Neo (Black et al., 2021) and GPT-J (Wang & Komatsuzaki, 2021), which are trained on The Pile (Gao et al., 2020), a dataset containing text from a variety of sources as well as 8% GitHub code.”
Documenting Large Webtext Corpora: A Case Study on the Colossal Clean Crawled Corpus (2021, background): “Since then, other corpora have been (partially) constructed from Common Crawl, e.g., P ILE (Gao et al., 2020), CCN ET (Wenzek et al., 2020), and M C4 (Xue et al., 2021).”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “OPT is trained on 180B unique tokens from a combination of the datasets used in RoBERTa (Liu et al., 2019), the Pile (Gao et al., 2020), and PushShift.io Reddit (Baumgartner et al., 2020; Roller et al., 2020) using a next-word prediction objective.”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “We use the following baseline models where appropriate (e.g. in settings where they support the language of the evaluation dataset): • mGPT (Shliazhko et al., 2022), GPT-style models trained on 60 languages from Wikipedia and Common Crawl • GPT-Neo (Black et al., 2021), GPT-J-6B (Wang and Komatsuzak”
OPT: Open Pre-trained Transformer Language Models (2022, methodology): “The pre-training corpus contains a concatenation of datasets used in RoBERTa (Liu et al., 2019b), the Pile (Gao et al., 2021a), and PushShift.io Red-dit (Baumgartner et al., 2020; Roller et al., 2021).”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “When the Pile was originally made, the only language model larger than GPT-NeoX-20B that 12https://discuss.huggingface.co/t/ bpe-tokenizers-and-spaces-before-words/475/2 13In private communication, the authors confirmed that Jurassic-1 was trained on the Pile (Gao et al., 2020).”
InCoder: A Generative Model for Code Infilling and Synthesis (2022, methodology): “The Pile [26], which was used to train recent generative code models that we compare to in Section 6, also contains these questions and answers but does not contain comments.”
CodeGen: An Open Large Language Model for Code with Multi-Turn Program Synthesis (2022, methodology): “For details on THEPILE, we refer to Gao et al. (2020).”