OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text

By Keiran Paster, Marco Dos Santos, Zhangir Azerbayev, Jimmy Ba. 2023

154 citations recorded by the source corpus.

There is growing evidence that pretraining on high quality, carefully thought-out tokens such as code or mathematics plays an important role in improving the reasoning abilities of large language models. For example, Minerva, a PaLM model finetuned on billions of tokens of mathematical documents from arXiv and the web, reported dramatically improved performance on problems that require quantitative reasoning. However, because all known open source web datasets employ preprocessing that does not faithfully preserve mathematical notation, the benefits of large scale training on quantitive web documents are unavailable to the research community. We introduce OpenWebMath, an open dataset inspired by these works containing 14.7B tokens of mathematical webpages from Common Crawl. We describe in detail our method for extracting text and LaTeX content and removing boilerplate from HTML documents, as well as our methods for quality filtering and deduplication. Additionally, we run small-scale experiments by training 1.4B parameter language models on OpenWebMath, showing that models trained on 14.7B tokens of our dataset surpass the performance of models trained on over 20x the amount of general language data. We hope that our dataset, openly released on the Hugging Face Hub, will help spur advances in the reasoning abilities of large language models.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Evaluation & Benchmarks

Citations in the corpus

Llemma: An Open Language Model For Mathematics (2023, methodology): “OpenWebMath filters CommonCrawl web pages based on math-related keywords and a classifier-based math score, preserves mathematical formatting (e.g., L A TEX, AsciiMath), and includes additional quality filters (e.g., perplexity, domain, length) and near-deduplication.”
Kimi k1.5: Scaling Reinforcement Learning with LLMs (2025, unknown): “The visual tower is initially trained in isolation without updating the language model parameters, then we unfreeze the language model layers, and ultimately increase the proportion of vision-text data”
Instruction Tuning for Large Language Models: A Survey (2023, unknown): “Furthermore, including a spectrum of complexity, from simple commands to multi-step reasoning problems, improves model robustness.”
Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking (2024, methodology): “We apply Quiet-STaR to Mistral 7B (Jiang et al., 2023) using the web text datasets OpenWebMath (Paster et al., 2023) and Colossal Clean Crawled Corpus (C4, Raffel et al. 2020).”
LoRA Learns Less and Forgets Less (2024, methodology): “The first regime is continued pretraining , which involves training on billions of unlabeled domain-specific tokens, most commonly via full finetuning; here we use the StarCoder-Python (Li et al., 2023a) and OpenWebMath (Paster et al., 2023) datasets (Table 1).”
MathPile: A Billion-Token-Scale Pretraining Corpus for Math (2023, unknown): “To improve models’ mathematical reasoning, pre-training corpora like AMPS [26] (despite a large-scale synthetic exercise set), ProofPile [4], and OpenWebMath [49] have been introduced, targeting various levels of mathematical problem-solving and theorem proving.”