Distilling Multi-Step Reasoning Capabilities of Large Language Models into Smaller Models via Semantic Decompositions

49 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Augmented Language Models: a Survey (2023, methodology): “As such, bootstrapping combines the data efficiency of few-shot prompting with some of the advantages of fine-tuning and can be successfully applied both to teach models to reason (Shridhar et al., 2022) and to use tools (Parisi et al.”
Teaching Small Language Models to Reason (2022, neutral): “In contrast to our work, Shridhar et al. (2022) focus on training two models, one for problem decomposition and one for solving.”
Specializing Smaller Language Models towards Multi-Step Reasoning (2023, background): “The actual practice of specialization is simply ﬁnetuning: to specialize a model towards a target ability, one simply tunes the model using the related data, which is the practice of concurrent work about smaller models’ CoT ability (Magister et al., 2022; Shridhar et al., 2022; Ho et al., 2022).”
Reasoning with Language Model Prompting: A Survey (2022, background): “CoT (Wei et al., 2022b), Self-C (Wang et al., 2022e), STaR (Zelikman et al., 2022), LogicSolver (Yang et al., 2022b), Least-to-Most (Zhang et al., 2022), ZeroCoT (Kojima et al., 2022), DIVERSE (Li et al., 2022d), Minerva (Lewkowycz et al., 2022), PROMPTPG (Lu et al., 2023b), Complexity (Fu et al., 2”
Towards Reasoning in Large Language Models: A Survey (2022, background): “There are also efforts to distill reasoning from LLMs into smaller models, such as the work by Li et al. (2022a); Shridhar et al. (2022); Magister et al. (2022).”