OPT: Open Pre-trained Transformer Language Models
By Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin and 9 more. 2022
4,844 citations recorded by the source corpus.
Large language models, which are often trained for hundreds of thousands of compute days, have shown remarkable capabilities for zero- and few-shot learning. Given their computational cost, these models are difficult to replicate without significant capital. For the few that are available through APIs, no access is granted to the full model weights, making them difficult to study. We present Open Pre-trained Transformers (OPT), a suite of decoder-only pre-trained transformers ranging from 125M to 175B parameters, which we aim to fully and responsibly share with interested researchers. We show that OPT-175B is comparable to GPT-3, while requiring only 1/7th the carbon footprint to develop. We are also releasing our logbook detailing the infrastructure challenges we faced, along with code for experimenting with all of the released models.
Subfields: Artificial Intelligence, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Yejin Choi
- Graham Neubig
- Noah A. Smith
- Yoshua Bengio
In Large Language Models
- Luke S. Zettlemoyer
- Furu Wei
- Yejin Choi
- Hinrich Schütze
- Mohit Bansal
- Noah A. Smith
- Timo Schick
- Graham Neubig
Citations in the corpus
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “Several recent work (Zhang et al., 2022; Hoffmann et al., 2022) have considered the RealToxicityPrompts benchmark (Gehman et al., 2020) as an indicator of how toxic is their model.”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, neutral): “, 2022), OPT(Zhang et al., 2022), and Falcon (Penedo et al.”
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023, neutral): “We additionally compare to OPT (66B) (Zhang et al., 2022) and the original davinci variant of GPT-3 (175B) (Brown et al., 2020), two models that are about 10 and 25 times larger than GPT-J.”
- Quantifying Memorization Across Neural Language Models (2022, methodology): “We finally study the OPT family of models (Zhang et al., 2022), that vary from 125 million to 175 billion parameters.”
- OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, methodology): “Following OPT (Zhang et al., 2022a), we will responsibly share versions of OPT-IML at both scales, and also release our OPT-IML Bench evaluation framework to facilitate future work in this direction.”
- BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “This is slightly less than the estimate made by the authors of the OPT paper, who stated that the total carbon footprint of their model is roughly 2 times higher due to experimentation, baselines and ablation (Zhang et al., 2022).”
- Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “They include but are not limited to generative models such as text-based language models (BLOOM [30], GPT-3 [40], OPT[271]), text-to-image models (ImaGen [205], DALL · E[170], Stable Diffusion [200]), and increasingly multimodal models [53] ( GPT-4 GPT-4 [ 171 ]).”
- The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, background): “We also note that recent popular open models (Zhang et al., 2022; Touvron et al., 2023) often indirectly leverage The Pile (Gao et al.”
- The Curse of Recursion: Training on Generated Data Makes Models Forget (2023, methodology): “Yet while current LLMs [Devlin et al., 2018, Liu et al., 2019, Brown et al., 2020, Zhang et al., 2022], including GPT-4 , were trained on predominantly human-generated text, this may change in the future.”
- AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models (2023, neutral): “Large Foundation Model: Recently, large foundation models, like LLMs (e.g., GPT-3 (Brown et al., 2020), GPT-4 (OpenAI, 2023), OPT (Zhang et al., 2022a) and FLAN-T5 (Chung et al., 2022)) have successfully demonstrated unprecedented performance in a wide range of natural language tasks.”
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “, 2022), and all models of such size class with fully public model parameters are decoder-only (Wang and Komatsuzaki, 2021; Le Scao et al., 2022; Zhang et al., 2022), the decision of which are often due to better hardware and software framework support.”
- Galactica: A Large Language Model for Science (2022, methodology): “We build on the open LLM training foundation they made possible with the OPT project (Zhang et al., 2022).”
- Large Language Models Are Human-Level Prompt Engineers (2022, methodology): “4 QUANTITATIVE ANALYSIS Can we use other LLMs for instruction proposal? We investigate other LLMs for instruction generation, including those with forward generation ability (OPT-175B (Zhang et al., 2022), OpenAI Codex (Chen et al.”
- Efficient Memory Management for Large Language Model Serving with PagedAttention (2023, methodology): “We use OPT [62] models with 13B, 66B, and 175B parameters and LLaMA [52] with 13B parameters for our evaluation.”
- Code Llama: Open Foundation Models for Code (2023, methodology): “…tasks include code summarization, refinement, translation (Rozière et al., 2020; 2021; Szafraniec et al., 2023) fixing bugs (Yasunaga & Liang, 2021; Zhang et al., 2022a; Prenner et al., 2022), fixing build errors (Tarlow et al., 2020) or generating unit tests (Tufano et al., 2020; Li et al., 2022;…”
- Fast Inference of Mixture-of-Experts Language Models with Offloading (2023, background): “However, the rapid scientific progress in this area would be impossible without open-access LLMs such as LLaMA 1 and 2 (Touvron et al., 2023), Falcon (TII UAE, 2023), BLOOM (Scao et al., 2022), OPT (Zhang et al., 2022), or NeoX/Pythia (Biderman et al., 2023).”
- Internet-augmented language models through few-shot prompting for open-domain question answering (2022, neutral): “(cid:44) → Question: Can an Arvanite Greek understand some of the Albanian Declaration of Independence?”
- Large Language Models are Zero-Shot Reasoners (2022, background): “For OPT, T0, GPT-J, GPT-Neo, and GPT-2, we used Hugging Face Transformer Library [Wolf et al., 2020].”