Llemma: An Open Language Model For Mathematics

By Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Qiaochu Jiang, Jia Deng, Stella Biderman, Sean Welleck. 2023

478 citations recorded by the source corpus.

We present Llemma, a large language model for mathematics. We continue pretraining Code Llama on the Proof-Pile-2, a mixture of scientific papers, web data containing mathematics, and mathematical code, yielding Llemma. On the MATH benchmark Llemma outperforms all known open base models, as well as the unreleased Minerva model suite on an equi-parameter basis. Moreover, Llemma is capable of tool use and formal theorem proving without any further finetuning. We openly release all artifacts, including 7 billion and 34 billion parameter models, the Proof-Pile-2, and code to replicate our experiments.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Information Systems and Management, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Information Systems and Management

In Large Language Models

Citations in the corpus

Mixtral of Experts (2024, background): “Figure 4 (Right) shows that the perplexity of Mixtral on a subset of the proof-pile dataset [2] decreases monotonically as the size of the context increases.”
DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “In the initial iteration, the classifier is trained using instances from OpenWebMath (Paster et al., 2023) as positive examples, while incorporating a diverse selection of other web pages to serve as negative examples.”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, unknown): “Llemma (Azerbayev et al., 2023) and MathPile (Wang et al., 2023c) continue pretraining LLMs with math corpus to improve domain capacity.”
A Survey on Knowledge Distillation of Large Language Models (2024, background): “The application of LLMs within the sub-domain of mathematics heralds a transformative era in mathematical research, education, and problem-solving (Azerbayev et al., 2023; Yu et al., 2023b).”
MAmmoTH2: Scaling Instructions from the Web (2024, unknown): “In contrast to previous math-centric approaches [Paster et al., 2023, Wang et al., 2023c, Shao et al., 2024], we aim for broad coverage of disciplines such as math, science, engineering, etc.”
Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement (2024, background): “…existing endeavors pre-training to large-scale, specialized LLMs focused on mathematics (Shao et al., 2024; Ying et al., 2024; Lewkowycz et al., 2022a; Azerbayev et al., 2024) have unequivocally demonstrated the value of extracting a considerable corpus of mathematical texts from digital databases.”
LLMSTEP: LLM proofstep suggestions in Lean (2023, background): “Active areas of work include fast CPU inference, improved models [3], and tasks beyond tactic prediction.”
TinyLlama: An Open-Source Small Language Model (2024, methodology): “The second corpus combined code and mathematical content, leveraging integrations with Starcoder (Li et al., 2023) and Proof Pile 2 (Azerbayev et al., 2023).”
Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotations (2023, methodology): “Improving and eliciting mathematical reasoning abilities of LLMs. Mathematical reasoning tasks are one of the most challenging tasks for LLMs. Researchers have proposed various methods to improve or elicit the mathematical reasoning ability of LLMs, which can be broadly divided into three groups: 1)”
MathPile: A Billion-Token-Scale Pretraining Corpus for Math (2023, unknown): “ProofPile-2 [5], encompassing OpenWebMath and others, is excluded from this comparison.”