Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizes

By Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander J. Ratner, Ranjay Krishna, Chen-Yu Lee, Tomas Pfister. 2023

990 citations recorded by the source corpus.

Deploying large language models (LLMs) is challenging because they are memory inefficient and compute-intensive for practical applications. In reaction, researchers train smaller task-specific models by either finetuning with human labels or distilling using LLM-generated labels. However, finetuning and distillation require large amounts of training data to achieve comparable performance to LLMs. We introduce Distilling step-by-step, a new mechanism that (a) trains smaller models that outperform LLMs, and (b) achieves so by leveraging less training data needed by finetuning or distillation. Our method extracts LLM rationales as additional supervision for training small models within a multi-task framework. We present three findings across 4 NLP benchmarks: First, compared to both finetuning and distillation, our mechanism achieves better performance with much fewer labeled/unlabeled training examples. Second, compared to few-shot prompted LLMs, we achieve better performance using substantially smaller model sizes. Third, we reduce both the model size and the amount of data required to outperform LLMs; our finetuned 770M T5 model outperforms the few-shot prompted 540B PaLM model using only 80% of available data on a benchmark, whereas standard finetuning the same T5 model struggles to match even by using 100% of the dataset. We release the code at: https://github.com/google-research/distilling-step-by-step .

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Efficiency & Compression

Citations in the corpus

The False Promise of Imitating Proprietary LLMs (2023, background): “A surge of recent publications have attempted to both locally imitate proprietary models for specific tasks (Sun et al., 2023; Hsieh et al., 2023; Honovich et al., 2022) and broadly imitate models, e.g., Alpaca (Taori et al., 2023), Vicuna (Chiang et al., 2023), Koala (Geng et al., 2023), GPT4ALL…”
Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution (2023, background): “Furthermore, prompting can improve model distillation (Wang et al., 2023c; Hsieh et al., 2023) and it can be used to simulate agentic behavior (Wang et al.”
Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking (2024, methodology): “…Qiao et al., 2024), or to use specific tools or memory (Yao et al., 2022; Lanchantin et al., 2024; Schick et al., 2024), while others investigate how one may distill the reasoning from a large language model into a smaller language model Ho et al. (2022); Li et al. (2022); Hsieh et al. (2023).”
Making Large Language Models Better Reasoners with Alignment (2023, positive): “Recent works (Chung et al., 2022; Hsieh et al., 2023; Mukherjee et al., 2023) find that training LLMs using data with a chain of thought (COT) reasoning process is a very effective method to improve the reasoning ability of LLMs.”
LlaVA-CoT: Let Vision Language Models Reason Step-By-Step (2024, positive): “Some vision-language models enhance visual reasoning by optimizing the visual encoding strategy [23, 31, 34] to produce cognition-focused visual tokens.”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “A commonly used instruction type for guiding labeling is chain-of-thought (CoT) prompt (Hsieh et al., 2023; Fu et al., 2023; Magister et al., 2023).”
Large Language Models as Commonsense Knowledge for Large-Scale Task Planning (2023, methodology): “There are also various ways to enhance runtime performance like using smaller LLMs like Llama [35, 36] or distilling LLM’s knowledge into a smaller model [30, 16, 23].”
Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (2023, background): “While they are not accessible for closed-box LFM’s7, recent work [17] demonstrates that richer signals like LFM rationales can help close the gap for task-specific distillation.”
On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes (2023, methodology): “This idea was quickly adapted to KD, by extending the teacher dataset with CoT prompts for fine-tuning the student (Magister et al., 2022; Ho et al., 2022; Hsieh et al., 2023).”
Impossible Distillation for Paraphrasing and Summarization: How to Make High-quality Lemonade out of Small, Low-quality Model (2023, background): “…with Language Model Another line of related works propose to directly distill models with LM-generated data, improving model reasoning (Zelikman et al., 2022; Hsieh et al., 2023), robustness (Chen et al., 2022a), controllability (Sclar et al., 2022), and language understanding (Ye et al., 2022).”
ReasonIR: Training Retrievers for Reasoning Tasks (2025, unknown): “Large language models (LLMs) can be effectively prompted to generate synthetic training data (Hsieh et al., 2023; Wang et al., 2022).”