Large Language Models Are Reasoning Teachers

By Namgyu Ho, Laura Schmid, Se-Young Yun. 2022

531 citations recorded by the source corpus.

Recent works have shown that chain-of-thought (CoT) prompting can elicit language models to solve complex reasoning tasks, step-by-step. However, prompt-based CoT methods are dependent on very large models such as GPT-3 175B which are prohibitive to deploy at scale. In this paper, we use these large models as reasoning teachers to enable complex reasoning in smaller models and reduce model size requirements by several orders of magnitude. We propose Fine-tune-CoT, a method that generates reasoning samples from very large teacher models to fine-tune smaller models. We evaluate our method on a wide range of public models and complex tasks. We find that Fine-tune-CoT enables substantial reasoning capability in small models, far outperforming prompt-based baselines and even the teacher model in many tasks. Additionally, we extend our method by leveraging the teacher model’s ability to generate multiple distinct rationales for each original sample. Enriching the fine-tuning data with such diverse reasoning results in a substantial performance boost across datasets, even for very small models. We conduct ablations and sample studies to understand the emergence of reasoning capabilities of student models. Our code implementation and data are available at https://github.com/itsnamgyu/reasoning-teacher.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Augmented Language Models: a Survey (2023, background): “Further, several instruction fine-tuning approaches (Ouyang et al., 2022; Chung et al., 2022; Iyer et al., 2022; Ho et al., 2022) use chain-of-thought style prompts to achieve remarkable improvements on popular benchmarks such as BBH (Srivastava et al., 2022) and MMLU (Hendrycks et al., 2021).”
Large Language Models Can Self-Improve (2022, positive): “Recent works (Magister et al., 2022; dhar et al., 2023; Ho et al., 2023) demonstrated that distillation on explanations generated from large models can increase the reasoning abilities of smaller models with ground truth filtering.”
Multimodal Chain-of-Thought Reasoning in Language Models (2023, positive): “However, there is severe information loss in the captioning process; thus, using the captions (as opposed to vision features) may suffer from a lack of mutual synergy in the representation space of different modalities.”
Teaching Small Language Models to Reason (2022, methodology): “Subsequently, Chung et al. (2022) explore finetuning a FLAN-based (Wei et al., 2021) Ho et al. (2022) and Li et al. (2022) also explore knowledge distillation with the difference of proposing diverse sampling and rationalization prompting, respectively.”
A Survey on Knowledge Distillation of Large Language Models (2024, background): “…results (Li et al., 2024b; Wang et al., 2023b), and reasoning tasks utilize LLMs for labeling Chains of Thought (CoT) explanations (Hsieh et al., 2023; Li et al., 2022; Ho et al., 2023; Magister et al., 2023; Fu et al., 2023; Ramnath et al., 2023; Li et al., 2023d; Liu et al., 2023g), among others.”
Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking (2024, background): “…Qiao et al., 2024), or to use specific tools or memory (Yao et al., 2022; Lanchantin et al., 2024; Schick et al., 2024), while others investigate how one may distill the reasoning from a large language model into a smaller language model Ho et al. (2022); Li et al. (2022); Hsieh et al. (2023).”
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (2023, methodology): “For example, Finetune-CoT [25] samples multiple reasoning paths from LLMs and finetune the student model with correct ones, while Self-Improve [27] chooses the one with the highest confidence.”
Reasoning with Language Model Prompting: A Survey (2022, background): “CoT (Wei et al., 2022b), Self-C (Wang et al., 2022e), STaR (Zelikman et al., 2022), Least-to-Most (Zhang et al., 2022), ZeroCoT (Kojima et al., 2022), Decomposed (Khot et al., 2023), Auto-CoT (Zhang et al., 2023b), PAL (Gao et al., 2022), Teaching (Magister et al., 2022), Fine-tune-CoT (Ho et al., 2”
Specializing Smaller Language Models towards Multi-Step Reasoning (2023, background): “The actual practice of specialization is simply finetuning: to specialize a model towards a target ability, one simply tunes the model using the related data, which is the practice of concurrent work about smaller models’ CoT ability (Magister et al., 2022; Shridhar et al., 2022; Ho et al., 2022).”
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023, methodology): “In fact, similar results have also been observed in (Wiegreffe et al., 2021; Magister et al., 2022; Ho et al., 2022) that simply treating rationale and”
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2022, background): “Given the recent surge of interest (Tay et al., 2023; Magister et al., 2022; Ho et al., 2022), however, smaller LMs will likely increasingly acquire such ability, making IRCoT compatible with many more LMs. IRCoT also relies on the base LM to support long inputs as multiple retrieved paragraphs…”
Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs (2024, unknown): “We adopt the settings for LLM reasoning tasks and primarily utilize the self-training strategy (Zelikman et al., 2022; Ho et al., 2023), where the model itself generates the training data.”
Prometheus: Inducing Fine-grained Evaluation Capability in Language Models (2023, background): “Similar to Chain-of-Thought Finetuning (Ho et al., 2022; Kim et al., 2023a), we fine-tune to sequentially generate the feedback and then the score.”
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (2023, methodology): “This idea was quickly adapted to KD, by extending the teacher dataset with CoT prompts for fine-tuning the student (Magister et al., 2022; Ho et al., 2022; Hsieh et al., 2023).”
CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society (2023, methodology): “…instruction or prompting methods , such as Chain-of-Thought (CoT) [123], zero-shot-CoT [61], and ReAct [126], have recently been developed to enhance the performance of LLMs on reasoning, arithmetic and decision making tasks [134, 118, 52, 73, 31, 103, 43, 64, 132, 46, 133, 105, 128, 25, 81, 109].”
A Survey on Model Compression for Large Language Models (2023, unknown): “These compressed LLMs, as demonstrated in prior studies [Frantar and Alistarh, 2023; Liu et al. , 2023; Ho et al. , 2023], continue to exhibit a significant performance gap when compared to their uncompressed counterparts.”
Large Language Model Alignment: A Survey (2023, background): “can enhance the reasoning ability of weaker models (Ho et al., 2022; Magister et al., 2022; Wei et al., 2022), resulting in advanced performance across various downstream tasks.”
Large Language Models are Better Reasoners with Self-Verification (2022, background): “We think that larger language models are able to generate more robust and accurate results in in-context learning Ho et al. (2022); Wang et al. (2023a), while smaller models are prone to generate erroneous text during generation, resulting in a lack of self-verification capabilities.”