Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster

By Nolan Dey, G. Gosal, Zhiming Chen, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, Joel Hestness. 2023

141 citations recorded by the source corpus.

We study recent research advances that improve large language models through efficient pre-training and scaling, and open datasets and tools. We combine these advances to introduce Cerebras-GPT, a family of open compute-optimal language models scaled from 111M to 13B parameters. We train Cerebras-GPT models on the Eleuther Pile dataset following DeepMind Chinchilla scaling rules for efficient pre-training (highest accuracy for a given compute budget). We characterize the predictable power-law scaling and compare Cerebras-GPT with other publicly-available models to show all Cerebras-GPT models have state-of-the-art training efficiency on both pre-training and downstream objectives. We describe our learnings including how Maximal Update Parameterization ($\mu$P) can further improve large model scaling, improving accuracy and hyperparameter predictability at scale. We release our pre-trained models and code, making this paper the first open and reproducible work comparing compute-optimal model scaling to models trained on fixed dataset sizes. Cerebras-GPT models are available on HuggingFace: https://huggingface.co/cerebras.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “…performance for 1B parameters models trained on 30GT; • core : comparisons with a wide range of models, notably based on the tasks reported in (Dey et al., 2023); • main : tasks available in the GPT-3 and PaLM papers (Brown et al., 2020; Chowdhery et al., 2022); • ext : tasks available in the…”
Emergent and Predictable Memorization in Large Language Models (2023, methodology): “…of writing, the only publicly-available pretrained LLM scaling suites trained on fully public training data are EleutherAI’s GPT-Neo [Black et al., 2021, Wang and Komatsuzaki, 2021, Black et al., 2022] and Pythia models [Biderman et al., 2023], and Cerebras systems’ Cerebras-GPT [Dey et al., 2023].”
The MiniPile Challenge for Data-Efficient Language Models (2023, background): “The Pile [17] is an 825GB dataset of diverse text, which has gained a lot of popularity in large language model research [34, 48, 10, 22, 13, 8].”
PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (2023, unknown): “7B [26], OPT-7B [27], and Pythia-6.”
MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies (2024, methodology): “In terms of training stability, the Tensor Program series (Yang et al., 2022; 2023) is introduced to ensure optimal hyper-parameter consistency across varying model scales, a technique employed in training CerebrasGPT (Dey et al., 2023).”
Will we run out of data? Limits of LLM scaling based on human-generated data (2022, background): “…as the biggest ones on our list. demand for larger training datasets, consistent with neural scaling laws for dense transformers which suggest that training data size should scale roughly with the square root of training compute (Hoffmann et al., 2022; Dey et al., 2023; Fetterman et al., 2023).”
mGPT: Few-Shot Learners Go Multilingual (2022, background): “The benefit of balancing the et al., 2022; Biderman et al., 2023; Dey et al., 2023), Chinese (Zeng et al., 2021), and Russian (Zmitrovich et al., 2023).”