GPT-4 Technical Report
By OpenAI Josh Achiam, OpenAI, Steven Adler, Josh Achiam, S. Agarwal, L. Ahmad, Ilge Akkaya, Florencia Leoni Aleman, D. Almeida, Janko Altenschmidt and 281 more. 2023
26,133 citations recorded by the source corpus.
We report the development of GPT-4, a large-scale, multimodal model which can accept image and text inputs and produce text outputs. While less capable than humans in many real-world scenarios, GPT-4 exhibits human-level performance on various professional and academic benchmarks, including passing a simulated bar exam with a score around the top 10% of test takers. GPT-4 is a Transformer-based model pre-trained to predict the next token in a document. The post-training alignment process results in improved performance on measures of factuality and adherence to desired behavior. A core component of this project was developing infrastructure and optimization methods that behave predictably across a wide range of scales. This allowed us to accurately predict some aspects of GPT-4's performance based on models trained with no more than 1/1,000th the compute of GPT-4.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models
View Paper: https://arxiv.org/abs/2303.08774
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Sergey Levine
- Percy Liang
- Yoshua Bengio
- Luke S. Zettlemoyer
- Quoc V. Le
- Yejin Choi
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Large Language Models
- Luke S. Zettlemoyer
- Yejin Choi
- Hinrich Schütze
- Noah A. Smith
- Jianfeng Gao
- Mohit Bansal
- Furu Wei
- Timo Schick
Citations in the corpus
- Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “They include but are not limited to generative models such as text-based language models (BLOOM [30], GPT-3 [40], OPT[271]), text-to-image models (ImaGen [205], DALL · E[170], Stable Diffusion [200]), and increasingly multimodal models [53] ( GPT-4 GPT-4 [ 171 ]).”
- Code Llama: Open Foundation Models for Code (2023, methodology): “ChatGPT and GPT-4 (OpenAI, 2023), PaLM (Chowdhery et al., 2022) and Chinchilla (Hoffmann et al., 2022) are closed source, while BLOOM (Scao et al., 2022), OPT (Zhang et al., 2022b), and the seminal work of Llama are public (Touvron et al., 2023a).”
- Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, background): “With the evolution of state-of-the-art commercial models and their applications towards very long context window lengths, such as 32k tokens (GPT-4 [25]) or even 100k (Claude [2]), the research question of efficient while accurate long context models is receiving increased attention.”
- QLoRA: Efficient Finetuning of Quantized LLMs (2023, neutral): “Baselines We compare our models to both research (Vicuna [10] and Open Assistant [31]) and commercial (GPT-4 [42], GPT-3.”
- StarCoder: may the source be with you! (2023, methodology): “OpenAI has published several technical reports of the GPT family of models (Brown et al., 2020; Chen et al., 2021; OpenAI, 2023a), showcasing the capabilities of their models.”
- LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention (2023, methodology): “For quantitative evaluation [10], we ask GPT-4 [45] to assess the response quality of instruction-following models on 80 questions Performance.”
- Foundation Models and Fair Use (2023, methodology): “Models such as GPT-3/4 (Brown et al., 2020; OpenAI, 2023), Stable Diffusion (Rombach et al., 2021), and Codex (Chen et al., 2021) are actively being integrated into a variety of products like Duolingo’s Language Learning App,3 Stability AI’s DreamStudio,4 GitHub’s CoPilot,5 and more.”
- GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models (2023, methodology): “…partly because out-of-the-box general-purpose LLMs may continue to be unreliable for various tasks due to issues such as factual inaccuracies, inherent biases, privacy concerns, and disinformation risks (Abid et al., 2021; Schramowski et al., 2022; Goldstein et al., 2023; OpenAI, 2023a).”
- C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models (2023, background): “While different LLMs may perform similarly in simple scenarios like casual conversations, complex tasks are often the key differentiators between LLMs (OpenAI, 2023).”
- UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, background): “The prosperity of LM techniques Radford et al. (2019); OpenAI (2023) significantly promotes progress in audio generation research in two directions.”
- SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, background): “the GPT series [1], [2] and DALL-E [30].”
- VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translation (2023, neutral): “Generative pre-training (GPT) (Radford et al., 2018) methods have demonstrated the great potential capability for few-shot and zero-shot problems in ChatGPT (OpenAI, 2022) and GPT-4 (OpenAI, 2023).”
- Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks (2022, neutral): “Our PoT+SC achieves the best-known results on all the evaluated MWP datasets and near best-known results on the financial datasets (excluding GPT-4 (OpenAI, 2023)).”
- Let's Verify Step by Step (2023, methodology): “All examples shown come from the large-scale generator (GPT-4).”