Datasheet for the Pile

By Stella Biderman, Kieran Bicheno, Leo Gao. 2022

49 citations recorded by the source corpus.

This datasheet describes the Pile, a 825 GiB dataset of human-authored text compiled by EleutherAI for use in large-scale language modeling. The Pile is comprised of 22 different text sources, ranging from original scrapes done for this project, to text data made available by the data owners, to third-party scrapes available online.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “(2020) that prioritize compilations of previously documented individual sources over crawled data provide a step towards addressing these issues (Biderman et al., 2022)).”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “Full details of the Pile can be found in the technical report (Gao et al., 2020) and the associated datasheet (Biderman et al., 2022).”
Foundation Models and Fair Use (2023, methodology): “, 2020) contains Books3, a dataset of copyrighted and commercially sold books downloaded from Bibliotik, a torrent tracker for books and learning materials (Presser, 2020; Biderman et al., 2022).”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, background): “The most notable example of this is the Pile, for which the authors themselves analyze and document a variety of syntactic and semantic properties of the dataset including structural statistics (n-gram counts, language, document sizes), topical distributions across its components, social bias and se”
Data Governance in the Age of Large-Scale Data-Driven Language Technology (2022, background): “There has also been an increased focus on analyzing other dimensions of data quality and stewardship [102, 107, 121, 123], with several noteworthy initiatives aiming to document both existing [9, 20, 30, 43], and newly developed [16, 60, 136] resources.”
You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings (2022, background): “GPT-J-6B EleutherAI Multinational English 6 B Open (Gao et al., 2020; Biderman et al., 2022)”
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024, methodology): “However, since we continue to SFT the model after the decay stage, we do not utilize the final checkpoints.”
Should ChatGPT be Biased? Challenges and Risks of Bias in Large Language Models (2023, methodology): “Another commonly-used dataset for language model training is The Pile [65, 66] an extensive and diverse collection of 22 smaller datasets, combining various sources of scientiﬁc articles, books, and web content.”
Emergent and Predictable Memorization in Large Language Models (2023, unknown): “All of these suites were trained on the Pile [Gao et al., 2020, Biderman et al., 2022].”
Eliciting Latent Predictions from Transformers with the Tuned Lens (2023, methodology): “Since BLOOM and GPT-2 do not have publicly available validation sets, we use the Pile validation set (Gao et al., 2020; Biderman et al., 2022).”