Self-Refine: Iterative Refinement with Self-Feedback

By Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang and 5 more. 2023

4,061 citations recorded by the source corpus.

Like humans, large language models (LLMs) do not always generate the best output on their first try. Motivated by how humans refine their written text, we introduce Self-Refine, an approach for improving initial outputs from LLMs through iterative feedback and refinement. The main idea is to generate an initial output using an LLMs; then, the same LLMs provides feedback for its output and uses it to refine itself, iteratively. Self-Refine does not require any supervised training data, additional training, or reinforcement learning, and instead uses a single LLM as the generator, refiner, and feedback provider. We evaluate Self-Refine across 7 diverse tasks, ranging from dialog response generation to mathematical reasoning, using state-of-the-art (GPT-3.5, ChatGPT, and GPT-4) LLMs. Across all evaluated tasks, outputs generated with Self-Refine are preferred by humans and automatic metrics over those generated with the same LLM using conventional one-step generation, improving by ~20% absolute on average in task performance. Our work demonstrates that even state-of-the-art LLMs like GPT-4 can be further improved at test time using our simple, standalone approach.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “, 2022), and follow-up instructions can be used to refine initial generations to make them more useful, engaging, and unbiased (Ganguli et al., 2023; Madaan et al., 2023).”
Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, background): “And for LMs, they can generate self-feedback (Madaan et al., 2023) or utilize a given feedback (Gou et al., 2023), like from classifiers, to self-correct the unsafe response.”
Instruction Tuning with GPT-4 (2023, background): “Meanwhile, recent studies show that GPT-4 is capable of identifying and fixing its own mistakes, and accurately judging the quality of responses(Peng et al., 2023; Bai et al., 2022; Madaan et al., 2023; Kim et al., 2023).”
Reasoning with Language Model is Planning with World Model (2023, methodology): “[38] directly reuse the same LLM to generate multi-aspect feedback and refine the previously generated output.”
DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023, background): “A growing body of work has revealed that, especially with instruction tuning (Ouyang et al., 2022), we can elicit sophisticated behavior via prompting (Wei et al., 2022; Wang et al., 2022b; Press et al., 2022; Yao et al., 2022; Khot et al., 2022; Madaan et al., 2023).”
A survey on large language model based autonomous agents (2023, neutral): “In fact, the action impact can encompass numerous instances, but for brevity, we only provide a few examples.”
Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution (2023, methodology): “Self-Refine (Madaan et al., 2023) prompts an LLM to generate a response, to provide feedback on the response, and to finally refine the solution.”
Improving Factuality and Reasoning in Language Models through Multiagent Debate (2023, neutral): “We find that our debate approach outperforms single model baselines such as zero-shot chain of thought (Kojima et al., 2022) and reflection (Shinn et al., 2023; Madaan et al., 2023) on a variety of six reasoning, factuality, and question-answering tasks.”
Tool Learning with Foundation Models (2023, positive): “…extrospective reasoning additionally considers feedback from the user and environment (Figure 5), and is thus better suited to complex tasks (Madaan et al., 2023; Wang et al., 2023b), such as multi-step QA and embodied learning, where decision-making at each step is dependent on the…”
Better Zero-Shot Reasoning with Role-Play Prompting (2023, methodology): “Self-refine (Madaan et al. 2023) generates an output through chain-of-thought, and then utilizes the same LLM to improve the initial output through iterative feedback and refinement.”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “Nonetheless, a branch of research (Madaan et al., 2023; Yao et al., 2023b; Xie et al., 2023) explores the potential of self-correction mechanisms, where an LLM corrects its initial responses using its built-in capabilities, independent of external feedback.”
RLAIF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, positive): “LLMs have also been extensively used for data generation (Wang et al., 2021b; Meng et al., 2023), augmentation (Feng et al., 2021) and in self-training setups (Wang et al., 2022b; Madaan et al., 2023).”
Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models (2023, background): “Another active research area is to incorporate external verification [39, 32] and human feedback [35] to allow LLMs to perform more reliably and align better with human preferences.”
A Survey on Large Language Models for Code Generation (2024, neutral): “CodeBLEU combines these elements (including weighted n-gram match, syntactic AST match, and semantic data-flow match) into a comprehensive evaluation metric.”
Language Models can Solve Computer Tasks (2023, methodology): “The most recent work that is in the same vein with RCI is Self-Refine [38] which uses localized and aspect-based feedback to iteratively refine outputs from LLMs.”