Measuring Progress on Scalable Oversight for Large Language Models

By Sam Bowman, Samuel R. Bowman, Jeeyoon Hyun, Ethan Perez, Edwin Chen, Edwin Chen, Craig Pettit, Scott Heiner, Kamilė Lukošiūtė, Amanda Askell and 38 more. 2022

226 citations recorded by the source corpus.

Developing safe and useful general-purpose AI systems will require us to make progress on scalable oversight: the problem of supervising systems that potentially outperform us on most skills relevant to the task at hand. Empirical work on this problem is not straightforward, since we do not yet have systems that broadly exceed our abilities. This paper discusses one of the major ways we think about this problem, with a focus on ways it can be studied empirically. We first present an experimental design centered on tasks for which human specialists succeed but unaided humans and current general AI systems fail. We then present a proof-of-concept experiment meant to demonstrate a key feature of this experimental design and show its viability with two question-answering tasks: MMLU and time-limited QuALITY. On these tasks, we find that human participants who interact with an unreliable large-language-model dialog assistant through chat -- a trivial baseline strategy for scalable oversight -- substantially outperform both the model alone and their own unaided performance. These results are an encouraging sign that scalable oversight will be tractable to study with present models and bolster recent findings that large language models can productively assist humans with difficult tasks.

Subfields: Artificial Intelligence, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Safety, Robustness & Interpretability

Citations in the corpus

Constitutional AI: Harmlessness from AI Feedback (2022, methodology): “There may also be ways for humans and AI systems to collaborate [Bowman et al., 2022] to provide better supervision than either can provide alone.”
Discovering Language Model Behaviors with Model-Written Evaluations (2022, background): “Our results suggest the importance of work on scalable oversight (Amodei et al., 2016; Saunders et al., 2022; Bowman et al., 2022), the problem of providing accurate supervision to AI systems to solve tasks that humans alone cannot easily supervise.”
GPQA: A Graduate-Level Google-Proof Q&A Benchmark (2023, neutral): “A similar effort to ours was undertaken in QuALITY (Pang et al., 2022), a crowdsourced reading comprehension dataset which has been used for scalable oversight experiments with open-ended dialogue (Bowman et al., 2022) and debate (Michael et al., 2023).”
GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, methodology): “…LLMs remain vulnerable to adversarial inputs that can elicit undesired behavior (Gehman et al., 2020; Bommasani et al., 2021; walkerspider, 2022; Perez et al., 2022; Perez & Ribeiro, 2022; Kang et al., 2023; Li et al., 2023; Ganguli et al., 2022; Schulhoff et al., 2023; OpenAI, 2023b; Jones et…”
Improving Code Generation by Training with Natural Language Feedback (2023, background): “More broadly, this approach also represents a weak version of scalable oversight (Bowman et al., 2022), in that model overseers can improve a model merely by evaluating its outputs, without manually generating new demonstrations, in a way that takes advantage of the capabilities that are being…”
The TRIPOD-LLM reporting guideline for studies using large language models (2025, unknown): “Human insight and oversight are critical components of the TRIPOD-LLM statement, reflecting an emphasis on components eventually critical for the responsible deployment of LLMs (although deployment reliability and observability are outside the scope of this paper) 42–44 .”
The alignment problem from a deep learning perspective (2022, methodology): “To address this, RLHF has been used to train policies to assist human supervisors, e.g. by critiquing the main policy’s outputs in natural language (albeit with mixed results thus far) [Saunders et al., 2022, Parrish et al., 2022b,a, Bowman et al., 2022, Bai et al., 2022b].”
Measuring Faithfulness in Chain-of-Thought Reasoning (2023, methodology): “Prompting and Sampling For each question on each task, we use the prompt shown in Table 1, modiﬁed slightly from Bowman et al. (2022).”
Toward expert-level medical question answering with large language models (2025, unknown): “These results indicate that, as the field progresses toward physician-level performance, improved evaluation frameworks (including highly specialized human raters) and work on scalable oversight 33 will be crucial for further measuring progress and aligning models.”
Towards Understanding Sycophancy in Language Models (2023, unknown): “6b, but here consider a 52B parameter helpful-only AI assistant, which was previously analyzed in Perez et al. (2022).”
Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond (2023, background): “implications of aligning such advanced systems, as they may present unique complexities and ethical concerns [8, 15].”
Large Language Model Alignment: A Survey (2023, background): “• Sandwiching (Bowman et al., 2022): Compared to the previous two paradigms, sandwiching operates on a different plane.”