Teaching Small Language Models to Reason

By Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, Aliaksei Severyn. 2022

417 citations recorded by the source corpus.

Chain of thought prompting successfully improves the reasoning capabilities of large language models, achieving state of the art results on a range of datasets. However, these reasoning capabilities only appear to emerge in models with at least tens of billions of parameters. In this paper, we explore the transfer of such reasoning capabilities to smaller models via knowledge distillation, also investigating model and dataset size trade-off. Specifically, we finetune a student model on the chain of thought outputs generated by a larger teacher model. Our experiments show that the proposed method improves task performance across arithmetic, commonsense and symbolic reasoning datasets. For example, the accuracy of T5 XXL on GSM8K improves from 8.11% to 21.99% and 18.42% when finetuned on PaLM 540B and GPT-3 175B generated chains of thought, respectively.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Large Language Models Can Self-Improve (2022, positive): “Recent works (Magister et al., 2022; dhar et al., 2023; Ho et al., 2023) demonstrated that distillation on explanations generated from large models can increase the reasoning abilities of smaller models with ground truth filtering.”
Multimodal Chain-of-Thought Reasoning in Language Models (2023, neutral): “Magister et al. (2022) and Ho et al. (2022) employed knowledge distillation by ﬁne-tuning a student model on the chain-of-thought outputs generated by a larger teacher model.”
Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes (2023, methodology): “In fact, similar results have also been observed in (Wiegreffe et al., 2021; Magister et al., 2022; Ho et al., 2022) that simply treating rationale and”
A Survey for In-context Learning (2023, background): “Magister et al. (2022) showed that it is possible to distill the reasoning ability to small language models such as T5-XXL.”
LLM-Assisted Code Cleaning For Training Accurate Code Generators (2023, background): “Prior work has demonstrated considerable successes in improving reasoning in LLM S (Yue et al., 2023; Magister et al., 2022; Fu et al., 2023) by performing supervised learning on natural language reasoning or planning steps .”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “A commonly used instruction type for guiding labeling is chain-of-thought (CoT) prompt (Hsieh et al., 2023; Fu et al., 2023; Magister et al., 2023).”
Large Language Model Alignment: A Survey (2023, background): “can enhance the reasoning ability of weaker models (Ho et al., 2022; Magister et al., 2022; Wei et al., 2022), resulting in advanced performance across various downstream tasks.”
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (2023, background): “Magister et al. [45] investigates the transfer of reasoning capabilities via knowledge distillation.”
Reasoning with Language Model Prompting: A Survey (2022, background): “Recently, an intuitive approach has been proposed to transfer the reasoning capabilities of large LMs to smaller LMs via knowledge distillation (Magister et al., 2022).”
Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions (2022, background): “Given the recent surge of interest (Tay et al., 2023; Magister et al., 2022; Ho et al., 2022), however, smaller LMs will likely increasingly acquire such ability, making IRCoT compatible with many more LMs. IRCoT also relies on the base LM to support long inputs as multiple retrieved paragraphs…”
ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs (2023, background): “A recent line of work has explored student-teacher frameworks with the goal of distilling reasoning capabilities from a stronger teacher to a weaker student (Magister et al., 2023; Fu et al., 2023; Ho et al., 2023; Saha et al., 2023; Mukherjee et al., 2023).”
Specializing Smaller Language Models towards Multi-Step Reasoning (2023, background): “The actual practice of specialization is simply finetuning: to specialize a model towards a target ability, one simply tunes the model using the related data, which is the practice of concurrent work about smaller models’ CoT ability (Magister et al., 2022; Shridhar et al., 2022; Ho et al., 2022).”
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (2023, methodology): “This idea was quickly adapted to KD, by extending the teacher dataset with CoT prompts for fine-tuning the student (Magister et al., 2022; Ho et al., 2022; Hsieh et al., 2023).”
Towards Reasoning in Large Language Models: A Survey (2022, background): “There are also efforts to distill reasoning from LLMs into smaller models, such as the work by Li et al. (2022a); Shridhar et al. (2022); Magister et al. (2022).”