Predictability and Surprise in Large Generative Models

By Deep Ganguli, Danny Hernandez, Liane Lovitt, Nova Dassarma, T. Henighan, Andy Jones, Nicholas Joseph, John Kernion, Benjamin Mann, Amanda Askell and 20 more. 2022

365 citations recorded by the source corpus.

Large-scale pre-training has recently emerged as a technique for creating capable, general-purpose, generative models such as GPT-3, Megatron-Turing NLG, Gopher, and many others. In this paper, we highlight a counterintuitive property of such models and discuss the policy implications of this property. Namely, these generative models have a paradoxical combination of predictable loss on a broad training distribution (as embodied in their ”scaling laws”), and unpredictable specific capabilities, inputs, and outputs. We believe that the high-level predictability and appearance of useful capabilities drives rapid development of such models, while the unpredictable qualities make it difficult to anticipate the consequences of model deployment. We go through examples of how this combination can lead to socially harmful behavior with examples from the literature and real world observations, and we also perform two novel experiments to illustrate our point about harms from unpredictability. Furthermore, we analyze how these conflicting properties combine to give model developers various motivations for deploying these models, and challenges that can hinder deployment. We conclude with a list of possible interventions the AI community may take to increase the chance of these models having a beneficial impact. We intend for this paper to be useful to policymakers who want to understand and regulate AI systems, technologists who care about the potential policy impact of their work, funders who want to support work addressing these challenges, and academics who want to analyze, critique, and potentially develop large generative models.

Subfields: Artificial Intelligence, Safety Research, Large Language Models, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety Research

In Large Language Models

In Safety, Robustness & Interpretability

Citations in the corpus

Emergent Abilities of Large Language Models (2022, background): “As we continue to train ever-larger language models, lowering the scale threshold for emergent abilities will become more important for allowing research on such abilities to available to the community broadly (Bommasani et al., 2021; Ganguli et al., 2022; Liang et al., 2022).”
The Capacity for Moral Self-Correction in Large Language Models (2023, background): “We do not advocate for the use of language models for admissions decisions—rather we treat this as a proxy experiment for discrimination that parallels previous work [18, 23, 30, 33, 48].”
Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, background): “This is surprising because our previous work typically shows larger models tend to generate more toxic model responses [2, 22].”
Can language models learn from explanations in context? (2022, background): “, 2020), smooth changes in loss can lead to qualitative differences in specific behaviours (Ganguli et al., 2022)— “more is different” (Anderson, 1972).”
Inverse scaling can become U-shaped (2022, background): “Medium-size models can perform the distractor task, which hurts performance, whereas large models can ignore the distractor task and perform the true task, which can be seen as an emergent ability of scaling (Ganguli et al., 2022; Wei et al., 2022a).”
Foundation Models and Fair Use (2023, background): “The recent literature on data governance and curation discusses fair use in machine learning (Jernite et al., 2022; Paullada et al., 2021; Ganguli et al., 2022).”
Transcending Scaling Laws with 0.1% Extra Compute (2022, neutral): “Secondly, we did not want to over-index on scaling studies that only measure impact on upstream cross entropy [Hernandez et al., 2022] which claims that repeating data in small quantities could be dis-proportionally harmful.”
Larger language models do in-context learning differently (2023, background): “In this paper we have also focused on the effect of scaling on in-context learning, which relates to a nascent body of work showing that scaling language models leads to qualitatively-different behavior (Ganguli et al., 2022; Wei et al., 2022b; Srivastava et al., 2022).”
AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap (2023, background): “All of this contributes to “capability unpredictability” [56], the idea that an LLM’s capabilities cannot be fully anticipated, even by the model’s creators, until its behavior on certain input is observed.”
Are aligned neural networks adversarially aligned? (2023, background): “As large language model parameter count, training dataset size, and training duration have been increased, the models have been found to exhibit complex behaviors [Brown et al., 2020, Wei et al., 2022b, Ganguli et al., 2022].”
Visual Adversarial Examples Jailbreak Aligned Large Language Models (2023, background): “LLMs have open-ended outputs, rendering the complete evaluation of their potential harm a persistent challenge (Ganguli et al. 2022a).”
Language models show human-like content effects on reasoning (2022, background): “For example, while large language models exhibit emergent behaviors, including some abstract reasoning performance (Brown et al., 2020; Ganguli et al., 2022; Nye et al., 2021a; Kojima et al., 2022; Wei et al., 2022a), they have been criticized for inconsistencies in their abstract reasoning (e.”
LMentry: A Language Model Benchmark of Elementary Language Tasks (2022, background): “Furthermore, the next-token prediction objective, used to optimize per-plexity, does not always align with human behaviors, expectations, or values (Bordia and Bowman, 2019; Huang et al., 2020; Schick et al., 2021; Sheng et al., 2021; Ganguli et al., 2022a,b).”
Many-shot Jailbreaking (2024, background): “At the same time, we also expect models to continue to become increasingly capable of both good and bad outcomes (Ganguli et al., 2022).”