MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning

By Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, Wenhu Chen. 2023

596 citations recorded by the source corpus.

We introduce MAmmoTH, a series of open-source large language models (LLMs) specifically tailored for general math problem-solving. The MAmmoTH models are trained on MathInstruct, our meticulously curated instruction tuning dataset. MathInstruct is compiled from 13 math datasets with intermediate rationales, six of which have rationales newly curated by us. It presents a unique hybrid of chain-of-thought (CoT) and program-of-thought (PoT) rationales, and also ensures extensive coverage of diverse fields in math. The hybrid of CoT and PoT not only unleashes the potential of tool use but also allows different thought processes for different math problems. As a result, the MAmmoTH series substantially outperform existing open-source models on nine mathematical reasoning datasets across all scales with an average accuracy gain between 16% and 32%. Remarkably, our MAmmoTH-7B model reaches 33% on MATH (a competition-level dataset), which exceeds the best open-source 7B model (WizardMath) by 23%, and the MAmmoTH-34B model achieves 44% accuracy on MATH, even surpassing GPT-4's CoT result. Our work underscores the importance of diverse problem coverage and the use of hybrid rationales in developing superior math generalist models.

Subfields: Artificial Intelligence, Computer Science Applications, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Science Applications

In Prompting & Reasoning

Citations in the corpus

Llemma: An Open Language Model For Mathematics (2023, background): “A number of recent works focus on supervised finetuning on task-relevant (input, output) pairs (e.g.,Yu et al. (2023); Yue et al. (2023)).”
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “The total number of training examples is 776K. • English mathematical datasets : We annotate GSM8K and MATH problems with tool-integrated solutions, and adopt a subset of MathInstruct (Yue et al., 2023) along with the training set of Lila-OOD (Mishra et al., 2022) where problems are solved with CoT ”
The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning (2023, background): “Specifically, model tuning may still be necessary for tasks such as coding (Luo et al., 2023), mathematics (Yue et al., 2023), interactive agents (Yin et al., 2023), etc.”
Large Language Models for Mathematical Reasoning: Progresses and Challenges (2024, background): “Instruction tuning in pre-training also emerges as a significant factor (Yue et al., 2023).”
MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems? (2024, methodology): “In addition to foundational visual instruction-following datasets, both SPHINX-MoE and InternLM-XComposer2 extend their training regimes to include specialized mathematical problems that are either text-only or visual, such as MathQA [2], Geometry3K [43], and MathInstruct [64].”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, unknown): “Some models enhance mathematical reasoning by integrating python tools, such as TORA (Gou et al., 2023), MAmmoTH (Yue et al., 2023), Openmathinstruct (Toshniwal et al., 2024), NuminaMath (Li et al., 2024b) etc.”
What matters when building vision-language models? (2024, unknown): “The detailed performance of Idefics2 across each category of MMMU (Yue et al., 2024) is present in Table 16.”
ReTool: Reinforcement Learning for Strategic Tool Use in LLMs (2025, unknown): “Tool-integrated reasoning was first introduced to help LLMs solve computationally intensive mathematical problems with the integration of programming strategies [1, 6, 18, 32].”
SPHINX-X: Scaling Data and Parameters for a Family of Multi-modal Large Language Models (2024, unknown): “MetaMath (Yu et al., 2023a) and MathInstruct (Yue et al., 2023b) are high-quality mathematical datasets with reasoning process.”
Making Large Language Models Better Reasoners with Alignment (2023, unknown): “Recent works (Chung et al., 2022; Hsieh et al., 2023; Mukherjee et al., 2023; Yue et al., 2023) find that training LLMs using data with a chain of thought (COT) reasoning process is a very effective method to improve the reasoning ability of LLMs.”
ReFT: Reasoning with Reinforced Fine-Tuning (2024, background): “Both models are reported to have strong performance in solving math problems and are commonly adopted in recent literature on reasoning tasks (Yue et al., 2023; Luo et al., 2023).”
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (2023, background): “2) fine-tuning : The fine-tuning meth-ods (Yu et al., 2023b; Luo et al., 2023; Yue et al., 2023; Wang et al., 2023b; Gou et al., 2023) can also enhance the mathematical reasoning ability of LLMs.”
Training Large Language Models to Reason in a Continuous Latent Space (2024, unknown): “…prompting LLMs (Wei et al., 2022; Khot et al., 2022; Zhou et al., 2022), or training LLMs to generate reasoning chains, either with supervised finetuning (Yue et al., 2023; Yu et al., 2023) or reinforcement learning (Wang et al., 2024; Havrilla et al., 2024; Shao et al., 2024; Yu et al., 2024a).”
LMFlow: An Extensible Toolkit for Finetuning and Inference of Large Foundation Models (2023, background): “LMFlow provides a complete finetuning workflow for these four stages, supporting large language models’ personalized training with limited computing resources.”
Qwen Technical Report (2023, background): “Additionally, fine-tuning on math-related instruction-following datasets (Si et al., 2023; Yuan et al., 2023a; Luo et al., 2023a; Yue et al., 2023; Chern et al., 2023a; Yu et al., 2023), has also been effective and more cost-effective than math-specific pretraining.”
LIMO: Less is More for Reasoning (2025, unknown): “Post-training research focuses on curating large-scale instruction data to teach reasoning (Yue et al., 2023; 2024; Li et al., 2024a; Yu et al., 2024) by scaling questions and solutions.”