Generating Sequences by Learning to Self-Correct

By Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, Yejin Choi. 2022

352 citations recorded by the source corpus.

Sequence generation applications require satisfying semantic constraints, such as ensuring that programs are correct, using certain keywords, or avoiding undesirable content. Language models, whether fine-tuned or prompted with few-shot demonstrations, frequently violate these constraints, and lack a mechanism to iteratively revise their outputs. Moreover, some powerful language models are of extreme scale or inaccessible, making it inefficient, if not infeasible, to update their parameters for task-specific adaptation. We present Self-Correction, an approach that decouples an imperfect base generator (an off-the-shelf language model or supervised sequence-to-sequence model) from a separate corrector that learns to iteratively correct imperfect generations. To train the corrector, we propose an online training procedure that can use either scalar or natural language feedback on intermediate imperfect generations. We show that Self-Correction improves upon the base generator in three diverse generation tasks - mathematical program synthesis, lexically-constrained generation, and toxicity control - even when the corrector is much smaller than the base generator.

Subfields: Information Systems, Software, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Software

In Natural Language Processing

Citations in the corpus

Self-Refine: Iterative Refinement with Self-Feedback (2023, methodology): “Following Welleck et al. (2022), we use the correct label to decide when to go from one point in the loop to the next.”
Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, background): “Self-Correction (Welleck et al., 2023) decouples an imperfect base generator (model) from a separate corrector that learns to iteratively correct imperfect generations and demonstrates improvement over the base generator.”
CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing (2023, methodology): “, 2022b), external metrics (Welleck et al., 2023) or knowledge (Peng et al.”
Training Language Models to Self-Correct via Reinforcement Learning (2024, neutral): “…to perform meaningful intrinsic self-correction, ﬁne-tuning approaches require running multiple models during inference, such as a separate reﬁnement model (Havrilla et al., 2024b; Welleck et al., 2023), or rely on “teacher” supervision to guide the process of self-correction (Qu et al., 2024).”
Reasoning with Language Model is Planning with World Model (2023, result): “Similar to our reward formulation, recent works have explored self-evaluation approaches to provide feedback for intermediate steps (Welleck et al., 2022; Shinn et al., 2023; Paul et al., 2023).”
Language Models can Solve Computer Tasks (2023, methodology): “Following previous works [38, 62, 74], we use the correct label to decide when to stop the RCI loop.”
Teaching Large Language Models to Self-Debug (2023, background): “Another line of work trains a language model to refine the initial model outputs based on external feedback on prediction quality (Welleck et al., 2023; Liu et al., 2023), which improves the performance on several natural language and reasoning tasks.”
Faith and Fate: Limits of Transformers on Compositionality (2023, methodology): “Finally, we suggest augmenting Transformers with planning modules as well as using refinement methods, that can iteratively improve their generations [63, 36].”