GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow

By Sid Black, Leo Gao, Phil Wang, Connor Leahy, Stella Biderman. 2021

964 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

TruthfulQA: Measuring How Models Mimic Human Falsehoods (2021, background): “GPT-Neo/J (Black et al., 2021; Wang and Komatsuzaki, 2021) is a variant of GPT-3 with a different training set (Gao et al., 2020).”
Quantifying Memorization Across Neural Language Models (2022, methodology): “We primarily study the GPT-Neo model family (Black et al., 2021; Wang and Komatsuzaki, 2021) trained on the Pile dataset (Gao et al., 2020).”
Program Synthesis with Large Language Models (2021, background): “Hendrycks et al. [2021] evaluate large language models on this data, specifically finetuned GPT-2 [Radford et al., 2019] and GPT-Neo [Black et al., 2021], as well as few-shot prediction with GPT-3 [Brown et al., 2020].”
Deduplicating Training Data Makes Language Models Better (2021, background): “3B (Black et al., 2021) are part of a 50 token matches of its training data, the Pile (Gao et al.”
Evaluating Large Language Models Trained on Code (2021, methodology): “Two recent works similar in spirit to Codex are GPT-Neo (Black et al., 2021) and GPT-J (Wang & Komatsuzaki, 2021), which are trained on The Pile (Gao et al., 2020), a dataset containing text from a variety of sources as well as 8% GitHub code.”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “We use the following baseline models where appropriate (e.g. in settings where they support the language of the evaluation dataset): • mGPT (Shliazhko et al., 2022), GPT-style models trained on 60 languages from Wikipedia and Common Crawl • GPT-Neo (Black et al., 2021), GPT-J-6B (Wang and Komatsuzak”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, background): “7B parameters) (Black et al., 2021), GPT-J-6B (Wang and Komatsuzaki, 2021), Megatron-11B1, Pangu-α-13B (Zeng et al.”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “We compare with the following models: the GPT-3 series (Brown et al., 2020), the FairSeq series (Artetxe et al., 2021), the GPT-Neo(X)/J models (Black et al., 2021; Wang & Komatsuzaki, 2021; Black et al., 2022), the OPT series (Zhang et al., 2022), the BigScience Architecture and Scaling Pile model…”
ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection (2022, background): “While there are several approaches for prompting pretrained LLMs (Liu et al., 2021b), a recent and promising direction is demonstration-based prompting (Gao et al., 2021; Mishra et al., 2021).”
Code Llama: Open Foundation Models for Code (2023, methodology): “Early observations with LLMs such as GPT-Neo (Black et al., 2021) or GPT-J (Wang & Komatsuzaki, 2021) showed that adding code in the training data makes program synthesis possible even with medium size LLMs. Code from open-source software is now a standard part of the training data for general-purpo”
Large Language Models are Zero-Shot Reasoners (2022, methodology): “For OPT, T0, GPT-J, GPT-Neo, and GPT-2, we used Hugging Face Transformer Library [Wolf et al., 2020].”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, methodology): “The standard (duplicated) Pile is 334B tokens using the GPT-NeoX tokenizer, so some data in the Pile may not be seen by the standard Pythia models.”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “For the initial model we use GPT-Neo-125, a variant of GPT-Neo (Black et al., 2021) on Hugging-face Transformers (Wolf et al., 2020).”
Adversarial Training for High-Stakes Reliability (2022, positive): “7B [49] fine-tuned on our story dataset.”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, background): “While the performance of Frozen on VL benchmarks falls short compared to the state of the art, we feel the approach is promising due to its practicality, and the public availability of large, pretrained LMs such as GPT-J [54], PanGu-α [59], and GPT-Neo [3].”
A systematic evaluation of large language models of code (2022, methodology): “In code modeling, CodeGPT (124M) (Lu et al., 2021), CodeParrot (1.5B) (Tunstall et al., 2022), GPT-Neo (2.7B) (Black et al., 2021), GPT-J (6B) (Wang & Komatsuzaki, 2021), Codex (12B) (Chen et al., 2021), GPT-NeoX (20B) (Black et al., 2022), and Google’s (137B) (Austin et al., 2021) belong to this…”