MAmmoTH2: Scaling Instructions from the Web

By Xiang Yue, T. Zheng, Ge Zhang, Wenhu Chen. 2024

170 citations recorded by the source corpus.

Instruction tuning improves the reasoning abilities of large language models (LLMs), with data quality and scalability being the crucial factors. Most instruction tuning data come from human crowd-sourcing or GPT-4 distillation. We propose a paradigm to efficiently harvest 10 million naturally existing instruction data from the pre-training web corpus to enhance LLM reasoning. Our approach involves (1) recalling relevant documents, (2) extracting instruction-response pairs, and (3) refining the extracted pairs using open-source LLMs. Fine-tuning base LLMs on this dataset, we build MAmmoTH2 models, which significantly boost performance on reasoning benchmarks. Notably, MAmmoTH2-7B's (Mistral) performance increases from 11% to 36.7% on MATH and from 36% to 68.4% on GSM8K without training on any in-domain data. Further training MAmmoTH2 on public instruction tuning datasets yields MAmmoTH2-Plus, achieving state-of-the-art performance on several reasoning and chatbot benchmarks. Our work demonstrates how to harvest large-scale, high-quality instruction data without costly human annotation or GPT-4 distillation, providing a new paradigm for building better instruction tuning data.

Subfields: Computer Vision and Pattern Recognition, Large Language Models, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Vitaliy Chiley
2. M. Paul
3. Jingren Zhou
4. John P. Cunningham
5. David Lopez-Paz

In Large Language Models

1. Jingren Zhou

Citations in the corpus

• How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2023, unknown): “1”
• LIMO: Less is More for Reasoning (2025, unknown): “…LLMs can be effectively aligned with user preferences through relatively small amounts of instruction data (Zhou et al., 2023), teaching models to reason—particularly in mathematics and pro-gramming—is widely believed to require vastly more training examples (Paster et al., 2023; Yue et al., 2024).”
• MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (2024, unknown): “Our concurrent work, MAmmoTH2 [34], also identifies real-world user queries from the web.”
• LoRA Learns Less and Forgets Less (2024, methodology): “Other work develops Monte Carlo Tree Search methods to automatically supervise the intermediate reasoning steps while solving math problems (Luo et al., 2024), and Yue et al. (2024) generates questions and answers from the pretraining web corpus.”
• Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement (2024, methodology): “Beyond recalling a diverse set of mathematical documents and filtering out low-quality data, we draw inspiration from previous efforts in generating synthetic mathematical data (Yue et al., 2024; Zhou et al., 2024).”
• Process Reinforcement through Implicit Rewards (2025, unknown): “…used for aligning human preferences (Christiano et al., 2017; Ouyang et al., 2022; Cui et al., 2024), but the open-source community mostly adopt imitation learning methods (Yuan et al., 2024a; Yue et al., 2024; Wei et al., 2024; Liu et al., 2024) to enhance the reasoning capabilities of LLMs.”