Self-critiquing models for assisting human evaluators

By William Saunders, Jeff Wu, Steven Bills, Ouyang Long, Ouyang Long, Jonathan Ward, Jan Leike. 2022

423 citations recorded by the source corpus.

We fine-tune large language models to write natural language critiques (natural language critical comments) using behavioral cloning. On a topic-based summarization task, critiques written by our models help humans find flaws in summaries that they would have otherwise missed. Our models help find naturally occurring flaws in both model and human written summaries, and intentional flaws in summaries written by humans to be deliberately misleading. We study scaling properties of critiquing with both topic-based summarization and synthetic tasks. Larger models write more helpful critiques, and on most tasks, are better at self-critiquing, despite having harder-to-critique outputs. Larger models can also integrate their own selfcritiques as feedback, refining their own summaries into better ones. Finally, we motivate and introduce a framework for comparing critiquing ability to generation and discrimination ability. Our measurements suggest that even large models may still have relevant knowledge they cannot or do not articulate as critiques. These results are a proof of concept for using AI-assisted human feedback to scale the supervision of machine learning systems to tasks that are difficult for humans to evaluate directly. We release our training datasets, as well as samples from our critique assistance experiments.

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

Citations in the corpus

Constitutional AI: Harmlessness from AI Feedback (2022, methodology): “Similar work involving model self-critique and natural language feedback includes [Zhao et al., 2021, Scheurer et al., , Saunders et al., 2022]; their methods are very similar to our supervised constitutional step.”
Self-Refine: Iterative Refinement with Self-Feedback (2023, background): “…a variety of tasks, including summarization (Scheurer et al., 2022), script generation (Tandon et al., 2021), program synthesis (Le et al., 2022a; Yasunaga and Liang, 2020), and other tasks (Bai et al., 2022a; Schick et al., 2022b; Saunders et al., 2022a; Bai et al., 2022b; Welleck et al., 2022).”
Measuring Progress on Scalable Oversight for Large Language Models (2022, methodology): “…Self-Critique, Ampliﬁcation and Recursive Reward Modeling: In this family of approaches (Leike et al., 2018; Christiano et al., 2018; Wu et al., 2021; Saunders et al., 2022), loosely speaking, the human participants iteratively (i) supervise a model using a human-supervision-based technique like…”
When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, background): “Many recent works have studied how to align language models with human feedback (Nakano et al., 2021; Ouyang et al., 2022; Scheurer et al., 2022; Saunders et al., 2022; Schick et al., 2022).”
Improving alignment of dialogue agents via targeted human judgements (2022, background): “Initial evidence from one-step debate is mixed: Saunders et al. (2022) find that model-generated critiques help humans notice flaws in summaries, but in Parrish et al. (2022) accuracy did not improve when humans were shown explanations.”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “Similar work involving model self-critique and natural language feedback includes (Zhao et al., 2021; Scheurer et al., 2022; Saunders et al., 2022) f -divergence objectives for generative models In the literature, there have been several studies exploring the use of f -divergences in generative…”
BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage (2022, background): “…from deployment, including never-endinglearning from language data (Carlson et al., 2010), improving web search (Agichtein et al., 2006), the Dynabench system which evaluates a number of NLP tasks (Kiela et al., 2021), or learning from feedback to improve summarization (Saunders et al., 2022).”
Language Model Cascades (2022, neutral): “Saunders et al. (2022) study model generated critiques in the context of summarization.”
News Summarization and Evaluation in the Era of GPT-3 (2022, background): “, 2022) or in non-standard settings (Saunders et al., 2022).”
LLM Evaluators Recognize and Favor Their Own Generations (2024, neutral): “…2020), model-based benchmarks (Shashidhar et al., 2023; Zeng et al., 2023; Yuan et al., 2023; Fu et al., 2023; Li et al., 2024), self-refinement (Saunders et al., 2022; Madaan et al., 2023; Lee et al., 2023; Shridhar et al., 2023), and constitutional AI (Bai et al., 2022), LLMs are increasingly…”
RLTF: Reinforcement Learning from Unit Test Feedback (2023, background): “Studies have also shown that offline RL can face issues such as distribution shift (Fujimoto et al., 2019; Schrittwieser et al., 2021), leading to unstable training and suboptimal policy performance.”
Generative Verifiers: Reward Modeling as Next-Token Prediction (2024, neutral): “Posing verification as generating “yet another token” allows it to tap better into the generation capabilities of LLMs, by making it straight-forward to employ CoT reasoning and additional inference-time compute for better verification.”
AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, methodology): “However, there exist methods in the literature other than those explored in AlpacaFarm that can incorporate alternative sources of feedback such as natural language [74, 32, 22, 59, 55, 13, 56, 40], numeric ratings [44, 31], or execution traces [13].”
Training Language Models with Language Feedback at Scale (2023, background): “Saunders et al. (2022) show that LMs themselves write high-quality feedback on LM outputs.”
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, background): “The dynamics of collaborating or competing agents play a key role in determining the success of AI systems [6, 26, 27, 84, 99, 9, 10].”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, methodology): “Their potential to augment human intellect continues to burgeon (Saunders et al., 2022).”
Training Language Models to Self-Correct via Reinforcement Learning (2024, neutral): “Several works that go beyond prompting rely on ﬁne-tuning with demonstrations of revisions, e.g. obtaining revisions directly from human annotators (Saunders et al., 2022) or stronger models (Ye et al., 2023; Qu et al., 2024).”
ULTRAFEEDBACK: Boosting Language Models with Scaled AI Feedback (2023, positive): “In previous research, it is shown that advanced AI systems are capable of conducting chatbot evaluations (Dubois et al., 2023; Zheng et al., 2023a), giving textual critiques (Ye et al., 2023; Wang et al., 2023c), or assisting human annotators (Saunders et al., 2022).”