Large Language Models Can Self-Improve

By Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, Jiawei Han. 2022

879 citations recorded by the source corpus.

Large Language Models (LLMs) have achieved excellent performances in various tasks. However, fine-tuning an LLM requires extensive supervision. Human, on the other hand, may improve their reasoning abilities by self-thinking without external inputs. In this work, we demonstrate that an LLM is also capable of self-improving with only unlabeled datasets. We use a pre-trained LLM to generate"high-confidence"rationale-augmented answers for unlabeled questions using Chain-of-Thought prompting and self-consistency, and fine-tune the LLM using those self-generated solutions as target outputs. We show that our approach improves the general reasoning ability of a 540B-parameter LLM (74.4%->82.1% on GSM8K, 78.2%->83.0% on DROP, 90.0%->94.4% on OpenBookQA, and 63.4%->67.9% on ANLI-A3) and achieves state-of-the-art-level performance, without any ground truth label. We conduct ablation studies and show that fine-tuning on reasoning is critical for self-improvement.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Constitutional AI: Harmlessness from AI Feedback (2022, background): “Some other recent works on self-supervision include [Shi et al., 2022, Huang et al., 2022].”
Scaling Instruction-Finetuned Language Models (2022, background): “Perhaps the most related work here is Huang et al. (2022), which shows that reasoning performance improves by multi-task ﬁnetuning a model on several self-generated synthetic CoT datasets.”
Teaching Small Language Models to Reason (2022, background): “Lastly, Huang et al. (2022) present one related experiment, however, we present a more in-depth exploration on more datasets.”
RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback (2023, methodology): “We note that this experiment is not a strict example of “self-improvement” (Huang et al., 2022), as the AI labeler is the instruction-tuned PaLM 2 XS, whereas the initial policy is PaLM 2 XS ﬁne-tuned on Reddit TL;DR summarization.”
Specializing Smaller Language Models towards Multi-Step Reasoning (2023, methodology): “language model self-improvement (Huang et al., 2022) which also use CoT data augmentation, but they only consider large models and do not show the tradeoff between model abilities.”
Decomposition Enhances Reasoning via Self-Evaluation Guided Decoding (2023, neutral): “, 2022) and tuning on self-generations (Huang et al., 2022).”
Scaling Relationship on Learning Mathematical Reasoning with Large Language Models (2023, background): “Several works applied the idea of rejection sampling along with other techniques to filter the diverse sampled reasoning paths for fine-tuning data augmentation (Huang et al., 2022; Zelikman et al., 2022; Ni et al., 2023; Zhu et al., 2023).”
Honest Students from Untrusted Teachers: Learning an Interpretable Question-Answering Pipeline from a Pretrained Language Model (2022, background): “Finally, concurrent work shows that it is possible to distill from large pretrained language models us-ing self-consistency in chain-of-thought prompting, without any labels at all (Huang et al., 2022).”
From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024, neutral): “While some works leverage metrics or external tools for this feasibility assessment (Huang et al., 2023a; Yuan et al.)”
Distilling Reasoning Capabilities into Smaller Language Models (2022, methodology): “Huang et al. (2022) used the most voted outputs to further ﬁne-tune the model for better performance.”
Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking (2024, background): “Later work aimed to leverage additional information or assumptions such as Huang et al. (2022) which demonstrated that the algorithm proposed in STaR could still work if one assumed that the majority-vote answer was correct (although this has a lower ultimate performance).”
Large Language Models Are Reasoning Teachers (2022, methodology): “This is similar to Wang et al. (2022b); Ze-likman et al. (2022); Huang et al. (2022), where diverse reasoning paths are generated and marginalized to ﬁnd the optimal answer.”
Large language models in medicine (2023, background): “Although this limits potential applications to semi-autonomous AI, these could nevertheless revolutionize clinical work by automating some time-consuming cognitive labor 14 .”
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023, methodology): “also been used as additional finetuning data “selfimprove” LLMs (Zelikman et al., 2022; Huang et al., 2022).”
Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023, background): “The response demonstrating the highest degree of consistency across the various reasoning trajectories is more inclined to represent the accurate solution ([58]).”