Scaling Relationship on Learning Mathematical Reasoning with Large Language Models

By Zheng Yuan, Hongyi Yuan, Cheng Li, Guanting Dong, Chuanqi Tan, Chang Zhou. 2023

354 citations recorded by the source corpus.

Mathematical reasoning is a challenging task for large language models (LLMs), while the scaling relationship of it with respect to LLM capacity is under-explored. In this paper, we investigate how the pre-training loss, supervised data amount, and augmented data amount influence the reasoning performances of a supervised LLM. We find that pre-training loss is a better indicator of the model's performance than the model's parameter count. We apply supervised fine-tuning (SFT) with different amounts of supervised data and empirically find a log-linear relation between data amount and model performance, and we find better models improve less with enlarged supervised datasets. To augment more data samples for improving model performances without any human effort, we propose to apply Rejection sampling Fine-Tuning (RFT). RFT uses supervised models to generate and collect correct reasoning paths as augmented fine-tuning datasets. We find with augmented samples containing more distinct reasoning paths, RFT improves mathematical reasoning performance more for LLMs. We also find RFT brings more improvement for less performant LLMs. Furthermore, we combine rejection samples from multiple models which push LLaMA-7B to an accuracy of 49.3\% on GSM8K which outperforms the supervised fine-tuning (SFT) accuracy of 35.9\% significantly.

Subfields: Artificial Intelligence, Large Language Models, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Scaling & Learning Theory

Citations in the corpus

DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “• We provide a unified paradigm to understand different methods, such as RFT, DPO, PPO, and GRPO.”
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (2023, methodology): “There are also previous works that augment mathematical reasoning data for fine-tuning [43, 79].”
ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solving (2023, background): “Rejection sampling Fine-Tuning (RFT) leverages multiple models to generate diverse reasoning paths for fine-tuning (Yuan et al., 2023).”
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (2023, background): “…capabilities across various tasks (Park et al., 2023; Kaddour et al., 2023; Song et al.; Li et al., 2023a; Wang et al., 2023a; Chen et al., 2023; Zheng et al., 2023; Wang et al., 2023c), However, even the most advanced LLMs face challenges in complex multi-step mathematical reasoning problems…”
Making Large Language Models Better Reasoners with Alignment (2023, positive): “For instance, AFT exceeds the strongest baseline RFT by 0.88% in average accuracy.”
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (2024, methodology): “…39]; 2) improving the LLM proposal distribution by either applying targeted optimization on specific reasoning tasks by finetuning with RL [32, 35, 49, 50] enabling models to critique and revise their answers iteratively [4, 8, 23, 30]; 3) enabling LLMs to benefit from additional test-time…”
Qwen2 Technical Report (2024, methodology): “…in evaluations of both fundamental language capabilities and instruction-tuned functionalities Specifically, Qwen2-72B-Instruct, our instruction-tuned variant, scores 9.1 on MT-Bench (Zheng et al., 2023), 48.1 on Arena-Hard (Chiang et al., 2024), and 35.7 on LiveCodeBench (Jain et al., 2024).”
Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training (2023, neutral): “Rejection sampling is a simple and effective approach for finetuning data augmentation to improve LLMs’ ability on single/multiple task(s) such as multistep reasoning (Yuan et al., 2023a; Zelikman et al., 2022) and alignment with human preference (Dong et al., 2023; Bai et al., 2022).”
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models (2023, background): “All evaluations follow the settings from Google et al. (2023), unless specified otherwise.”
Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement (2024, methodology): “We adopt an iterative approach that leverages rejection sampling, guided by reward modeling and annotated answers, to incrementally enhance the quality of responses (Yuan et al., 2023).”
SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning (2024, neutral): “The primary methodology involves utilizing data generated either by the model itself or by alternative models, which is subsequently filtered through reward models or reward functions, and then employed for continued model training. are Rejection sampling Fine-Tuning[58] and distillation.”
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (2024, background): “In the context of supervised fine-tuning (SFT) of LLMs, human-crafted data has proven to be a remarkably effective source that enhances the performance of LLMs on tasks such as code generation (Roziere et al., 2023; Yang et al., 2023) and mathematical reasoning (Yuan et al., 2023; Luo et al., 2023).”
MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023, methodology): “For instance, both RFT and WizardMath can increase the accuracy on GSM8K (Cobbe et al., 2021) by 30%+, one of their fine-tuning datasets, but hurt the accuracy on out-of-domain datasets like MMLU-Math (Hendrycks et al., 2021a) or AQuA (Ling et al., 2017) by up to 10%.”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, neutral): “Figure 4 illustrates that the evolution process does not yield higher similarity scores.”
How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2023, methodology): “We collect three SFT datasets { D 1 , D 2 , D 3 } including GSM8K RFT (Yuan et al., 2023b), Code Alpaca (Chaudhary, 2023), and ShareGPT (Chiang et al., 2023) to represent math reasoning, coding, and general human-aligning ability SFT dataset respectively.”
rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (2025, neutral): “Although rejection sampling methods [Yuan et al., 2023, Brown et al., 2024] can improve data quality, they do not guarantee correct intermediate steps.”
Traj-CoA: Patient Trajectory Modeling via Chain-of-Agents for Lung Cancer Risk Prediction (2025, unknown): “To investigate if further training could enhance the predictive performance of Traj-CoA, we conducted a preliminary study using rejection sampling fine-tuning (RFT) [66].”