Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision

By Collin Burns, Pavel Izmailov, J. Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas R. Joglekar, Jan Leike and 3 more. 2023

504 citations recorded by the source corpus.

Widely used alignment techniques, such as reinforcement learning from human feedback (RLHF), rely on the ability of humans to supervise model behavior - for example, to evaluate whether a model faithfully followed instructions or generated safe outputs. However, future superhuman models will behave in complex ways too difficult for humans to reliably evaluate; humans will only be able to weakly supervise superhuman models. We study an analogy to this problem: can weak model supervision elicit the full capabilities of a much stronger model? We test this using a range of pretrained language models in the GPT-4 family on natural language processing (NLP), chess, and reward modeling tasks. We find that when we naively finetune strong pretrained models on labels generated by a weak model, they consistently perform better than their weak supervisors, a phenomenon we call weak-to-strong generalization. However, we are still far from recovering the full capabilities of strong models with naive finetuning alone, suggesting that techniques like RLHF may scale poorly to superhuman models without further work. We find that simple methods can often significantly improve weak-to-strong generalization: for example, when finetuning GPT-4 with a GPT-2-level supervisor and an auxiliary confidence loss, we can recover close to GPT-3.5-level performance on NLP tasks. Our results suggest that it is feasible to make empirical progress today on a fundamental challenge of aligning superhuman models.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Statistical Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohammad Shoeybi
2. Bryan Catanzaro
3. Tianyi Zhou
4. Yu Wu
5. Dacheng Tao
6. Wei Xiong
7. Can Xu
8. Hanze Dong

In Representation & Self-Supervised Learning

1. Chen Zhu
2. Tao Shen
3. Dacheng Tao
4. Tianyi Zhou
5. Mohammad Shoeybi
6. Tom Goldstein
7. Kaixuan Ji
8. Ming Li

In Statistical Learning Theory

1. Chen Zhu
2. Dacheng Tao
3. Tianyi Zhou
4. Tom Goldstein
5. Wei Xiong
6. Hanze Dong
7. Doyen Sahoo
8. Tongliang Liu

Citations in the corpus

• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “We believe such WEAK-TO-STRONG (Burns et al., 2023) alignment methods will bring a fundamental change to the learning algorithms.”
• The TRIPOD-LLM reporting guideline for studies using large language models (2025, unknown): “Human insight and oversight are critical components of the TRIPOD-LLM statement, reflecting an emphasis on components eventually critical for the responsible deployment of LLMs (although deployment reliability and observability are outside the scope of this paper) 42–44 .”
• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (2024, background): “The self-training algorithm (Vapnik, 1999; Grandvalet & Bengio, 2004; Lee, 2013) has also been proved to be able to convert weak learners to strong learners in mixture models without the need for additional labeled data (Frei et al., 2022; Kou et al., 2022).”
• Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models (2024, unknown): “…2024; Casper et al., 2023) that sophisticated LMs trained with human feedback (Christiano et al., 2023) in settings with easy-to-hard domain shift (Burns et al., 2023; Hase et al., 2024) will be misaligned because, in these settings, “overseer approval” and “desirable behavior” are equally…”
• A Survey on Knowledge Distillation of Large Language Models (2024, background): “However, Burns et al. (2023) find that achieving the full capabilities of strong models requires more than naive finetuning, suggesting the need for further research in this area.”
• ODIN: Disentangled Reward Mitigates Hacking in RLHF (2024, background): “Moreover, it has been observed that RLHF has weak-to-strong generalization, where the policy becomes more creative than the supervision it receives [Burns et al., 2023].”
• RLHF Workflow: From Reward Modeling to Online RLHF (2024, background): “x a 1 a 2 y x a 1 a 2 y x a 1 a 2 y Preference Data Prompt Responses Preference resulting policy model often performs poorly when faced with out-of-distribution data (Burns et al., 2023).”