MathPile: A Billion-Token-Scale Pretraining Corpus for Math
By Zengzhi Wang, Rui Xia, Pengfei Liu. 2023
77 citations recorded by the source corpus.
High-quality, large-scale corpora are the cornerstone of building foundation models. In this work, we introduce MathPile, a diverse and high-quality math-centric corpus comprising about 9.5 billion tokens. Throughout its creation, we adhered to the principle of"less is more", firmly believing in the supremacy of data quality over quantity, even in the pre-training phase. Our meticulous data collection and processing efforts included a complex suite of preprocessing, prefiltering, language identification, cleaning, filtering, and deduplication, ensuring the high quality of our corpus. Furthermore, we performed data contamination detection on downstream benchmark test sets to eliminate duplicates and conducted continual pre-training experiments, booting the performance on common mathematical reasoning benchmarks. We aim for our MathPile to boost language models' mathematical reasoning abilities and open-source its different versions and processing scripts to advance the field.
Subfields: Computational Theory and Mathematics, Computer Vision and Pattern Recognition, Theoretical Computer Science, Large Language Models, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
In Large Language Models
In Evaluation & Benchmarks
Citations in the corpus
- DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “5 7B (Guo et al., 2024), using arXiv corpora that underwent varied processing pipelines: • MathPile (Wang et al., 2023c): an 8.9B-token corpus developed with cleaning and filtering heuristic rules, over 85% of which are scientific arXiv papers; • ArXiv-RedPajama (Computer, 2023): the entirety of…”
- MAmmoTH2: Scaling Instructions from the Web (2024, unknown): “In contrast to previous math-centric approaches [Paster et al., 2023, Wang et al., 2023c, Shao et al., 2024], we aim for broad coverage of disciplines such as math, science, engineering, etc.”
- WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, unknown): “Llemma (Azerbayev et al., 2023) and MathPile (Wang et al., 2023c) continue pretraining LLMs with math corpus to improve domain capacity.”