STaR: Bootstrapping Reasoning With Reasoning

By Eric Zelikman, Yuhuai Wu, Noah D. Goodman. 2022

981 citations recorded by the source corpus.

Generating step-by-step"chain-of-thought"rationales improves language model performance on complex reasoning tasks like mathematics or commonsense question-answering. However, inducing language model rationale generation currently requires either constructing massive rationale datasets or sacrificing accuracy by using only few-shot inference. We propose a technique to iteratively leverage a small number of rationale examples and a large dataset without rationales, to bootstrap the ability to perform successively more complex reasoning. This technique, the"Self-Taught Reasoner"(STaR), relies on a simple loop: generate rationales to answer many questions, prompted with a few rationale examples; if the generated answers are wrong, try again to generate a rationale given the correct answer; fine-tune on all the rationales that ultimately yielded correct answers; repeat. We show that STaR significantly improves performance on multiple datasets compared to a model fine-tuned to directly predict final answers, and performs comparably to fine-tuning a 30$\times$ larger state-of-the-art language model on CommensenseQA. Thus, STaR lets a model improve itself by learning from its own generated reasoning.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Chain of Thought Prompting Elicits Reasoning in Large Language Models (2022, background): “, 2019, inter alia) or generates synthetic datasets (Talmor et al., 2020; Zelikman et al., 2022).”
Large Language Models Are Human-Level Prompt Engineers (2022, methodology): “Our approach to optimizing this prompt was inspired by Zelikman et al. (2022).”
Scaling Instruction-Finetuned Language Models (2022, background): “, 2022, inter alia) or focuses on models of substantially smaller scale (Ling et al., 2017; Camburu et al., 2018; Rajani et al., 2019; Talmor et al., 2020; Zelikman et al., 2022).”
Large Language Models are Zero-Shot Reasoners (2022, background): “…ability can be substantially increased by making them produce step-by-step reasoning, either by ﬁne-tuning [Rajani et al., 2019, Cobbe et al., 2021, Zelikman et al., 2022, Nye et al., 2022] or few-shot prompting [Wei et al., 2022, Wang et al., 2022, Chowdhery et al., 2022] (See Table 6 for…”
Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “…on curated data (Solaiman & Dennison, 2021; Ngo et al., 2021; Welbl et al., 2021; Chung et al., 2022) or selected samples from the model (Zelikman et al., 2022; Scheurer et al., 2022; Dohan et al., 2022), and fine-tuning the language model using reinforcement learning with a learned…”
Can language models learn from explanations in context? (2022, methodology): “C L ] 1 0 O ct 2 02 2 down the steps of a reasoning process for LMs can improve few-shot performance (Nye et al., 2021; Wei et al., 2022; Zelikman et al., 2022).”
Few-Shot Self-Rationalization with Natural Language Prompts (2021, methodology): “Zelikman et al. (2022) extend this approach by using explanations generated in a few-shot manner to reﬁne the same GPT-J (Wang and Komatsuzaki, 2021) model.”
Faithful Reasoning Using Large Language Models (2022, methodology): “For example, various methods of prompting (Wei et al., 2022) and iterative ﬁne-tuning (Zelikman et al., 2022) have been used to encourage models to produce reasoning traces, and while this has led to improvement of ﬁnal answer accuracy these traces do not support our understanding of how the answer…”
Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning (2022, background): “The authors found that the generated reasoning traces often contain unrelated or incorrect steps while still resulting in the correct answer (see examples in the appendices of (Wei et al., 2022; Zelikman et al., 2022)).”
Learning by Distilling Context (2022, methodology): “We deﬁne the input distribution D to be uniform over all possible 1 through 8 digit addition questions, identical to those used by Nye et al. (2021a); Zelikman et al. (2022).”
Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models (2023, background): “Further, recent studies [20, 47] have shown that LLMs can be self-improved by leveraging self-generated rationales under the zero-shot setting.”
Let's Verify Step by Step (2023, background): “Within the domain of logical reasoning, models trained with outcome supervision regularly use incorrect reasoning to reach the correct final answer (Zelikman et al., 2022; Creswell et al., 2022).”
Decomposed Prompting: A Modular Approach for Solving Complex Tasks (2022, background): “, 2022), coupling in-context learning with sparse fine-tuning (Zelikman et al., 2022), and even presented many of them under a single probabilistic programming framework (Dohan et al.”
Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training (2023, neutral): “Rejection sampling is a simple and effective approach for finetuning data augmentation to improve LLMs’ ability on single/multiple task(s) such as multistep reasoning (Yuan et al., 2023a; Zelikman et al., 2022) and alignment with human preference (Dong et al., 2023; Bai et al., 2022).”
Recommender Systems in the Era of Large Language Models (LLMs) (2023, neutral): “Current researches continue to delve into the application of CoT in LLMs, such as STaR [50], THOR [51], and Tab-CoT [52].”
Making Language Models Better Reasoners with Step-Aware Verifier (2022, neutral): “7This is motivated by Zelikman et al. (2022).”
Successive Prompting for Decomposing Complex Questions (2022, background): “Works like Chain-of-Thought reasoning (Wei et al., 2022; Zelikman et al., 2022) especially focus on compositional questions where they provide a chain of reasoning as demonstrations.”
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025, positive): “This stage typically falls into three main categories: supervised fine-tuning using human-curated or distilled data (Wang et al., 2023), self-improvement iteration (Zelikman et al., 2022; Gulcehre et al., 2023), and reinforcement learning (Ouyang et al., 2022).”
LLM-Assisted Code Cleaning For Training Accurate Code Generators (2023, background): “…finetuning – in common-sense reasoning (West et al., 2022), text-summarization (Sclar et al., 2022), mathematical reasoning (Luo et al., 2023a; Yue et al., 2023), tool use (Patil et al., 2023), coding (Luo et al., 2023b), and general-purpose reasoning Li et al. (2023b); Zelikman et al. (2022).”