MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models

By L. Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zheng Li, Adrian Weller, Weiyang Liu. 2023

702 citations recorded by the source corpus.

Large language models (LLMs) have pushed the limits of natural language understanding and exhibited excellent problem-solving ability. Despite the great success, most existing open-source LLMs (e.g., LLaMA-2) are still far away from satisfactory for solving mathematical problem due to the complex reasoning procedures. To bridge this gap, we propose MetaMath, a fine-tuned language model that specializes in mathematical reasoning. Specifically, we start by bootstrapping mathematical questions by rewriting the question from multiple perspectives without extra knowledge, which results in a new dataset called MetaMathQA. Then we fine-tune the LLaMA-2 models on MetaMathQA. Experimental results on two popular benchmarks (i.e., GSM8K and MATH) for mathematical reasoning demonstrate that MetaMath outperforms a suite of open-source LLMs by a significant margin. Our MetaMath-7B model achieves 66.4% on GSM8K and 19.4% on MATH, exceeding the state-of-the-art models of the same size by 11.5% and 8.7%. Particularly, MetaMath-70B achieves an accuracy of 82.3% on GSM8K, slightly better than GPT-3.5-Turbo. We release all the MetaMathQA dataset, the MetaMath models with different model sizes and the training code for public use.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

JudgeLM: Fine-tuned Large Language Models are Scalable Judges (2023, neutral): “However we think it is a common problem of large language models (Imani et al., 2023), and future advancement on base models (Yu et al., 2023a; Luo et al., 2023) would alleviate this problem.”
The rise and potential of large language model based agents: a survey (2025, positive): “To further improve the reasoning ability of LLMs or LLM-based agents, current research primarily explores supervised ﬁne-tuning (SFT) methods [152] and interactive training approaches [153,154].”
Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling (2024, neutral): “…(en) [20], CLEVR-Math (en) [144], Super-CLEVR (en) [141], MapQA (en) [23], MAVIS (en) [300], Geometry3K (en) [164], TallyQA (en) [2], MetaMath (en) [281], GEOS (en) [206], UniGeo (en) [26], Mathematics GeomVerse (en) [107], CMM-Math (zh) [154] ChartQA (en) [181], MMTab (en) [310], PlotQA (en)…”
MAmmoTH2: Scaling Instructions from the Web (2024, positive): “In contrast, the "Other" domain, which includes a diverse range of subjects, achieves the highest score on GSM8K, emphasizing the importance of diversity in the training data.”
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (2023, methodology): “Specifically, PRM800K is annotated based on the output from GPT-4, and consequently, a discrepancy arises for the output of open-source LLaMA models fine-tuned on MetaMATH.”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, neutral): “Particularly, WizardMath-Llama 70B achieves a substantial improvement of 10.5% on GSM8K and 32.0% on MATH compared to MetaMath-Llama 70B in testing accuracy.”
How far are we to GPT-4V? Closing the gap to commercial multimodal models with open-source suites (2024, neutral): “These datasets span multiple tasks, task ratio dataset Laion-EN (en) [89], Laion-ZH (zh) [89], COYO (zh) [12], Captioning 53.9% GRIT (zh) [86], COCO (en) [19], TextCaps (en) [95] Objects365 (en&zh) [93], GRIT (en&zh) [86], Detection 5.2% All-Seeing (en&zh) [114] Wukong-OCR (zh) [29], LaionCOCO-OCR (”
Scaling Synthetic Data Creation with 1,000,000,000 Personas (2024, background): “Representative studies include Wang et al. (2022) and Yu et al. (2023).”
rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinking (2025, positive): “Notable works include NuminaMath [Jia LI and Polu, 2024a] and MetaMath [Yu et al., 2023b].”
PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models (2024, positive): “Fine-tuning large language models (LLMs) is a highly effective technique for boosting their capabilities in various tasks [1, 2, 3, 4], ensuring models to follow instructions [5, 6, 7], and instilling models with desirable behaviors while eliminating undesirable ones [8, 9].”
LoRA Learns Less and Forgets Less (2024, methodology): “The MetaMathQA dataset (Yu et al. (2023), https://huggingface.co/datasets/meta-math/MetaMathQA ) contains 395,000 samples that are bootsrapped from the GSM (Cobbe et al., 2021) and Math (Hendrycks et al., 2021) training sets.”
A Survey on Evaluation of Large Language Models (2023, neutral): “These include tasks such as mathematical reasoning [214, 224, 230] and structured data inference [79, 141].”
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (2024, methodology): “Recent research has also highlighted the rephrasing capability of LLMs in prompting for better LLM response (Deng et al., 2023; Prasad et al., 2023) as well as augmenting synthetic data for more effective SFT (Yu et al., 2023; Liu et al., 2023).”
Qwen Technical Report (2023, background): “Additionally, fine-tuning on math-related instruction-following datasets (Si et al., 2023; Yuan et al., 2023a; Luo et al., 2023a; Yue et al., 2023; Chern et al., 2023a; Yu et al., 2023), has also been effective and more cost-effective than math-specific pretraining.”