Sparks of Artificial General Intelligence: Early experiments with GPT-4
By Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, J. Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundberg and 4 more. 2023
4,318 citations recorded by the source corpus.
Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition. The latest model developed by OpenAI, GPT-4, was trained using an unprecedented scale of compute and data. In this paper, we report on our investigation of an early version of GPT-4, when it was still in active development by OpenAI. We contend that (this early version of) GPT-4 is part of a new cohort of LLMs (along with ChatGPT and Google's PaLM for example) that exhibit more general intelligence than previous AI models. We discuss the rising capabilities and implications of these models. We demonstrate that, beyond its mastery of language, GPT-4 can solve novel and difficult tasks that span mathematics, coding, vision, medicine, law, psychology and more, without needing any special prompting. Moreover, in all of these tasks, GPT-4's performance is strikingly close to human-level performance, and often vastly surpasses prior models such as ChatGPT. Given the breadth and depth of GPT-4's capabilities, we believe that it could reasonably be viewed as an early (yet still incomplete) version of an artificial general intelligence (AGI) system. In our exploration of GPT-4, we put special emphasis on discovering its limitations, and we discuss the challenges ahead for advancing towards deeper and more comprehensive versions of AGI, including the possible need for pursuing a new paradigm that moves beyond next-word prediction. We conclude with reflections on societal influences of the recent technological leap and future research directions.
Subfields: Artificial Intelligence, Health Informatics, Large Language Models
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Yoshua Bengio
- Luke S. Zettlemoyer
- Quoc V. Le
- Yejin Choi
- Graham Neubig
In Health Informatics
- Max Tegmark
- Heng Ji
- Christopher D. Manning
- Jiawei Han
- Matei A. Zaharia
- Xiang Ren
- Gero Strobel
- Meredith Ringel Morris
In Large Language Models
Citations in the corpus
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, background): “However, their performance on downstream tasks and alignment with user intent can be significantly improved by fine-tuning on datasets of instructions and human-written completions [23, 36, 13, 39].”
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “This approach has been tried in our earlier blog post [7] and other concurrent or follow-up work [5, 25, 12, 11, 43, 15].”
- Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, critical): “Based on our new results, it is not clear that the newer generation of models have achieved neural Theory of Mind, corroborating findings by Ullman (2023), (Lenci, 2023), and Marcus and Davis (2023) and debunking claims of “emergence of TOM in LLMs” by Kosinski (2023) and Bubeck et al. (2023).”
- Let's Verify Step by Step (2023, background): “However, even state-of-the-art models are prone to producing falsehoods — they exhibit a tendency to invent facts in moments of uncertainty (Bubeck et al., 2023).”
- MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoning (2023, background): “As demonstrated in the example, for specialist domains that demand considerable expert knowledge such as the medical domain knowledge, employing a CoT approach might sometimes lead to hallucination (Bubeck et al., 2023; Guerreiro et al., 2023; Ji et al., 2023; Maynez et al., 2020).”
- RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval (2024, neutral): “…encoded within their parameters (Petroni et al., 2019; Jiang et al., 2020; Talmor et al., 2020; Rae et al., 2021; Hoffmann et al., 2022; Chowdhery et al., 2022; Bubeck et al., 2023; Kandpal et al., 2023) and models can be further improved with fine-tuning on downstream tasks (Roberts et al., 2020).”
- Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action (2023, background): “Inspired by the success of language models as general-purpose text processing systems [20, 122, 177], there has been a recent wave of multimodal systems trying to achieve similar general-purpose capabilities with additional modalities.”
- Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf (2023, background): “Furthermore, studies have shown that LLMs exhibit a certain degree of theory of mind capabilities (Bubeck et al., 2023; Shapira et al., 2023; Kosinski, 2023), as well as the potential to simulate believable human behaviors (Park et al., 2023).”
- Understanding the Effects of Iterative Prompting on Truthfulness (2024, background): “The advent and rapid evolution of Large Language Models (LLMs) represent a profound shift in the artificial intelligence landscape [1, 2].”
- A Simple and Effective Pruning Approach for Large Language Models (2023, background): “Large language models (Brown et al., 2020; OpenAI, 2023) have recently reshaped the field of NLP with their remarkable performance across a range of complex language benchmarks (Bommarito & Katz, 2022; Wei et al., 2022a; Bubeck et al., 2023).”
- CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, background): “These techniques underpin the impressive capabilities of recent dialogue LLMs [106, 116, 36, 9, 82, 13], which aim to simulate human-like conversations and provide personalized and interactive experiences for users, exhibiting the behavior of conversational AI agents [33].”
- LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion (2023, background): “Large language models (LLMs) have shown impressive performance in diverse tasks, primarily due to their capacity to follow instructions and access extensive, high-quality data, showing a promising future for artificial general intelligence (Bubeck et al., 2023).”
- Causal Reasoning and Large Language Models: Opening a New Frontier for Causality (2023, background): “Now, LLMs may have the capability to automate or assist with every step of this process and seamlessly transition bteween covariance-and logic-based causality. soning (Bubeck et al., 2023; Zhong et al., 2023; Ghazal et al., 2017).”
- LiveBench: A Challenging, Contamination-Free LLM Benchmark (2024, background): “Evaluating the mathematical abilities of LLMs has been one of the cornerstones of recent research in LLMs, featuring prominently in many releases and reports [6, 7, 42, 46].”
- How is ChatGPT's behavior changing over time? (2023, background): “p We acknowledge that the specific benchmark datasets used here does not comprehensively cover the complex behaviors of ChatGPT. d How LLMs perform on code generation is also of major interests (Bubeck et al., 2023; M. Chen et al., 2021).”
- Solving olympiad geometry without human demonstrations (2024, methodology): “25) can be considered to be in this category.”
- Exploring the Potential of Large Language Models (LLMs)in Learning on Graphs (2023, background): “Exemplars such as ChatGPT [17] and GPT4 [18], equipped with hundreds of billions of parameters, exhibit superior performance [19] on numerous text-related tasks from various domains.”
- Segment Everything Everywhere All at Once (2023, background): “The success of Large Language Models (LLMs) such as ChatGPT [38] have shown the importance of modern AI models in interacting with humans, and have provided a glimpse of AGI [3].”