GPT-NeoX-20B: An Open-Source Autoregressive Language Model

By Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang and 7 more. 2022

1,029 citations recorded by the source corpus.

We introduce GPT-NeoX-20B, a 20 billion parameter autoregressive language model trained on the Pile, whose weights will be made freely and openly available to the public through a permissive license. It is, to the best of our knowledge, the largest dense autoregressive model that has publicly available weights at the time of submission. In this work, we describe GPT-NeoX-20B’s architecture and training, and evaluate its performance. We open-source the training and evaluation code, as well as the model weights, at https://github.com/EleutherAI/gpt-neox.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

LLaMA: Open and Efficient Foundation Language Models (2023, background): “Rotary Embeddings [GPTNeo].”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “The Codex model is a language model that was finetuned on code, while the GPT models (Black et al., 2021; Wang and Komatsuzaki, 2021; Black et al., 2022) are trained on a mix of code and text like BLOOM.”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “…GPT-3 series (Brown et al., 2020), the FairSeq series (Artetxe et al., 2021), the GPT-Neo(X)/J models (Black et al., 2021; Wang & Komatsuzaki, 2021; Black et al., 2022), the OPT series (Zhang et al., 2022), the BigScience Architecture and Scaling Pile model (Scao et al., 2022b), PaLM-8B (Chowdhery…”
Galactica: A Large Language Model for Science (2022, background): “Models are trainedwith self-supervision on large, general corpuses and they performwell on hundreds of tasks (Brown et al., 2020; Rae et al., 2021; Hoffmann et al., 2022; Black et al., 2022; Zhang et al., 2022; Chowdhery et al., 2022).”
Code Llama: Open Foundation Models for Code (2023, methodology): “In parallel, models specifically trained or fine-tuned for code understanding and program synthesis from natural language prompts emerged with LLMs such as Codex (Chen et al., 2021), CodeT5 (Wang et al., 2021), InCoder (Fried et al., 2023), AlphaCode (Li et al., 2022), CodeGen (Nijkamp et al., 2023b”
Llemma: An Open Language Model For Mathematics (2023, neutral): “We hope that L LEMMA and Proof - Pile - 2 will be a useful base for future work on understanding language model generalization and dataset composition, investigating the limits of domain-specific language models, using language models as tools for mathematicians, and improving the mathematical…”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “This dataset has three major benefits over its competitors: first, it is freely and publicly available; second, it reports a higher downstream performance (Le Scao et al., 2022) than popular crawl-based datasets C4 (Raffel et al., 2020; Dodge et al., 2021) and OSCAR (Su´arez et al., 2019); and third”
StarCoder: may the source be with you! (2023, methodology): “Similarly, EleutherAI, a grassroots-turned-nonproﬁt research initiative, has released open-access LLMs including GPT-NeoX (Black et al., 2022), GPT-J (Wang & Komatsuzaki, 2021), and Pythia (Biderman et al., 2023), as well as the associated training data (Gao et al., 2021a).”
The case for 4-bit precision: k-bit Inference Scaling Laws (2022, methodology): “Large Language Models (LLMs) are widely adopted for zero/few-shot inference (Zhang et al., 2022; Black et al., 2022; Zeng et al., 2022; Scao et al., 2022), but they can be challenging to use both due to their large memory footprints – up to 352 GB of GPU memory for 175B models – and high latency.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, background): “, 2019), 10B-scale language models (Raffel et al., 2020; Shoeybi et al., 2019; Black et al., 2022), to 100Bscale GPT-3 (Brown et al.”
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, background): “Notably, many similarly capable models have already been released publicly in the past [Raffel et al., 2020, Wang and Komatsuzaki, 2021, Sanh et al., 2021, Black et al., 2022].”
MultiPL-E: A Scalable and Polyglot Approach to Benchmarking Neural Code Generation (2023, background): “neural language models (LLMs) trained for text generation have demonstrated the ability to perform program completion [5], [14], [15].”
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023, methodology): “GPT-NeoX [4] ! ! 20B The Pile Multiple Text: 730GB Code: 96GB (473B tokens trained) % % HumanEval”
The Gradient of Generative AI Release: Methods and Considerations (2023, background): “EleutherAI is a decentralized collective also prioritizes transparency and has released all system components, as seen in their GPT-J [99] and GPT-Neo [12] language models and the Pile dataset [33].”
A systematic evaluation of large language models of code (2022, methodology): “…modeling, CodeGPT (124M) (Lu et al., 2021), CodeParrot (1.5B) (Tunstall et al., 2022), GPT-Neo (2.7B) (Black et al., 2021), GPT-J (6B) (Wang & Komatsuzaki, 2021), Codex (12B) (Chen et al., 2021), GPT-NeoX (20B) (Black et al., 2022), and Google’s (137B) (Austin et al., 2021) belong to this category.”
Mass-Editing Memory in a Transformer (2022, methodology): “Although large autoregressive language models (Radford et al., 2019; Brown et al., 2020; Wang & Komatsuzaki, 2021; Black et al., 2022) are capable of recalling an impressive array of common facts such as “Tim Cook is the CEO of Apple” or “Polaris is in the constellation Ursa Minor” (Petroni et al.,…”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, methodology): “…for language modeling that has become widely used for training state-of-the-art English-language models (Lieber et al., 2021; Smith et al., 2022; Black et al., 2022; Zhang et al., 2022), and C4 and mC4 (Raffel et al., 2020; Xue et al., 2020), which have powered the T5 family of models; CC100…”