Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling

By Stella Biderman, Hailey Schoelkopf, Quentin Anthony, Herbie Bradley, Kyle O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff and 3 more. 2023

2,009 citations recorded by the source corpus.

How do large language models (LLMs) develop and evolve over the course of training? How do these patterns change as models scale? To answer these questions, we introduce \textit{Pythia}, a suite of 16 LLMs all trained on public data seen in the exact same order and ranging in size from 70M to 12B parameters. We provide public access to 154 checkpoints for each one of the 16 models, alongside tools to download and reconstruct their exact training dataloaders for further study. We intend \textit{Pythia} to facilitate research in many areas, and we present several case studies including novel results in memorization, term frequency effects on few-shot performance, and reducing gender bias. We demonstrate that this highly controlled setup can be used to yield novel insights toward LLMs and their training dynamics. Trained models, analysis code, training code, and training data can be found at \url{https://github.com/EleutherAI/pythia}.

Subfields: Artificial Intelligence, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Evaluation & Benchmarks

Citations in the corpus

OpenAssistant Conversations - Democratizing Large Language Model Alignment (2023, methodology): “Although models have for some time exhibited an extraordinary ability to ﬁt the training data and generalize based on their trained objective [2, 3], their adoption among the general public has until recently been slow.”
Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, background): “However, the rapid scientific progress in this area would be impossible without open-access LLMs such as LLaMA 1 and 2 (Touvron et al., 2023), Falcon (TII UAE, 2023), BLOOM (Scao et al., 2022), OPT (Zhang et al., 2022), or NeoX/Pythia (Biderman et al., 2023).”
QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “To additionally study the advantages of NF4 over other 4-bit data types, we use the setup of Dettmers and Zettlemoyer [13] and measure post-quantization zero-shot accuracy and perplexity across different models (OPT [73], LLaMA [58], BLOOM [53], Pythia [7]) for model sizes 125m - 13B.”
StarCoder: may the source be with you! (2023, background): “Similarly, EleutherAI, a grassroots-turned-nonproﬁt research initiative, has released open-access LLMs including GPT-NeoX (Black et al., 2022), GPT-J (Wang & Komatsuzaki, 2021), and Pythia (Biderman et al., 2023), as well as the associated training data (Gao et al., 2021a).”
Emergent and Predictable Memorization in Large Language Models (2023, background): “…of writing, the only publicly-available pretrained LLM scaling suites trained on fully public training data are EleutherAI’s GPT-Neo [Black et al., 2021, Wang and Komatsuzaki, 2021, Black et al., 2022] and Pythia models [Biderman et al., 2023], and Cerebras systems’ Cerebras-GPT [Dey et al., 2023].”
Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors (2022, neutral): “Even repeatedly retraining or finetuning smaller models [3] quickly costs too much for most practitioners.”
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, background): “While most pre-training has focused on English (Devlin et al., 2019; Rad-ford et al., 2019; Raffel et al., 2020; Biderman et al., 2023), there has also been considerable work focused on mono-lingual pre-training outside of English (Faysse et al., 2024; Gutiérrez-Fandiño et al., 2021; Zeng et al.,…”
Sparse Autoencoders Find Highly Interpretable Features in Language Models (2023, neutral): “In our case, the data vectors { x i } n vec i =1 are internal activations of a language model, such as Pythia-70M (Biderman et al., 2023), and { g j } n gt j =1 are unknown, ground truth network features.”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, background): “10 Dolly11 is Pythia 12B (Biderman et al., 2023) fine-tuned on DataBricks Dolly, human-written data created by Databricks.”
Quantifying Contamination in Evaluating Code Generation Capabilities of Language Models (2024, positive): “C ODER D ATA (Li et al., 2023), as well as three model series trained on either corpora, StarCoder-Base (Li et al., 2023), Pythia (Biderman et al., 2023) and CodeGen-NL (Nijkamp et al., 2023).”
∞Bench: Extending Long Context Evaluation Beyond 100K Tokens (2024, background): “LLMs are typically trained on sequences containing no more than 8K tokens (Tou-vron et al., 2023; Penedo et al., 2023; Biderman et al., 2023), and thus cannot well handle contexts exceeding 8K tokens.”
RWKV: Reinventing RNNs for the Transformer Era (2023, background): “RWKV even out-performs Pythia and GPT-Neo (Black et al., 2022) in four tasks: PIQA, OBQA, ARC-E, and COPA (See details in Appendix H).”
A Simple and Effective Pruning Approach for Large Language Models (2023, methodology): “In addition to LLaMA and LLaMA-2, we experiment with three previous LLM model families: namely OPT (Zhang et al., 2022), BLOOM (Scao et al., 2022) and Pythia (Biderman et al., 2023).”
MURI: High-Quality Instruction Tuning Datasets for Low-Resource Languages via Reverse Instructions (2024, neutral): “In summary, while instruction tuning has greatly advanced the capabilities of LLMs in following complex instructions, challenges remain in dataset diversity, validity, and the integration of multilingual content.”
How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources (2023, methodology): “We examine the impact of using different base models in Table 4, comparing LL A M A , OPT [54], and Pythia [4] models of comparable size trained on the Human+GPT data mix.”