Discovering Language Model Behaviors with Model-Written Evaluations

By Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu and 55 more. Annual Meeting of the Association for Computational Linguistics, 2022

917 citations recorded by the source corpus.

As language models (LMs) scale, they develop many novel behaviors, good and bad, exacerbating the need to evaluate how they behave. Prior work creates evaluations with crowdwork (which is time-consuming and expensive) or existing data sources (which are not always available). Here, we automatically generate evaluations with LMs. We explore approaches with varying amounts of human effort, from instructing LMs to write yes/no questions to making complex Winogender schemas with multiple stages of LM-based generation and filtering. Crowdworkers rate the examples as highly relevant and agree with 90-100% of labels, sometimes more so than corresponding human-written datasets. We generate 154 datasets and discover new cases of inverse scaling where LMs get worse with size. Larger LMs repeat back a dialog user's preferred answer ("sycophancy") and express greater desire to pursue concerning goals like resource acquisition and goal preservation. We also find some of the first examples of inverse scaling in RL from Human Feedback (RLHF), where more RLHF makes LMs worse. For example, RLHF makes LMs express stronger political views (on gun rights and immigration) and a greater desire to avoid shut down. Overall, LM-written evaluations are high-quality and let us quickly discover many novel LM behaviors.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

View Paper: https://arxiv.org/abs/2212.09251

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

OpenAssistant Conversations - Democratizing Large Language Model Alignment (2023, methodology): “While methods such as RLHF can mitigate some of these shortcomings, they may exacerbate others [26, 20].”
GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023, neutral): “In settings where annotators cannot do this, we would expect issues like hallucination (Zhang et al., 2023) and sycophancy (Perez et al., 2022b; Sharma et al., 2023) to be exacerbated.”
Many-shot Jailbreaking (2024, positive): “Our malevolent personality evaluations are based on the dataset accompanying the work of Perez et al. (2022b), which can be found in the following URL: https://github.com/anthropics/evals/tree/main/persona .”
XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2023, methodology): “This risk motivates a growing body of research that aims to make models safer, through techniques such as red-teaming (Ganguli et al., 2022; Perez et al., 2022a) and large-scale Figure 1: An example of exaggerated safety behaviour by the original llama-2-70b-chat-hf (Touvron et al., 2023), in…”
Role play with large language models (2023, background): “Consequently, if prompted with human-like dialogue, we shouldn’t be surprised if an agent role-plays a human character with all those human attributes, including the instinct for survival 22 .”
Steering Llama 2 via Contrastive Activation Addition (2023, methodology): “We then evaluate their steering effects on 50 held-out multiple-choice questions with the same format as our generation sets.”
Harms from Increasingly Agentic Algorithmic Systems (2023, background): “[145] show that increased training of a LLM with”
Towards Measuring the Representation of Subjective Global Opinions in Language Models (2023, neutral): “Additional examples where the model generates differing responses with CP versus LP are provided in Appendix C (Tables 12 and 13).”
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, methodology): “The evaluation aspects of our work are related to a growing line of work on simulating human annotation for evaluation [14, 51, 14, 50, 35, 36].”
Managing extreme AI risks amid rapid progress (2023, background): “They are unlikely to resolve automatically as AI systems get more capable 33,41–45 , and require dedicated research and engineering efforts.”
LLM Censorship: A Machine Learning Challenge or a Computer Security Problem? (2023, background): “However, extant defences have been empirically bypassed [38, 37, 28, 29, 41, 17, 50], and theoretical work [55] suggests that there will exist inputs to LLMs that elicit misaligned behaviour.”
Empowering Biomedical Discovery with AI Agents (2024, neutral): “For instance, integrating human feedback can promote desirable agent behavior, but it can also exacerbate persuasive abilities, echoing false beliefs [182].”