LeanDojo: Theorem Proving with Retrieval-Augmented Language Models
By Kaiyu Yang, Aidan M. Swope, Alex Gu, Rahul Chalamala, Peiyang Song, Shixing Yu, Saad Godil, Ryan J. Prenger, Anima Anandkumar. 2023
514 citations recorded by the source corpus.
Large language models (LLMs) have shown promise in proving formal theorems using proof assistants such as Lean. However, existing methods are difficult to reproduce or build on, due to private code, data, and large compute requirements. This has created substantial barriers to research on machine learning methods for theorem proving. This paper removes these barriers by introducing LeanDojo: an open-source Lean playground consisting of toolkits, data, models, and benchmarks. LeanDojo extracts data from Lean and enables interaction with the proof environment programmatically. It contains fine-grained annotations of premises in proofs, providing valuable data for premise selection: a key bottleneck in theorem proving. Using this data, we develop ReProver (Retrieval-Augmented Prover): an LLM-based prover augmented with retrieval for selecting premises from a vast math library. It is inexpensive and needs only one GPU week of training. Our retriever leverages LeanDojo's program analysis capability to identify accessible premises and hard negative examples, which makes retrieval much more effective. Furthermore, we construct a new benchmark consisting of 98,734 theorems and proofs extracted from Lean's math library. It features challenging data split requiring the prover to generalize to theorems relying on novel premises that are never used in training. We use this benchmark for training and evaluation, and experimental results demonstrate the effectiveness of ReProver over non-retrieval baselines and GPT-4. We thus provide the first set of open-source LLM-based theorem provers without any proprietary datasets and release it under a permissive MIT license to facilitate further research.
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Retrieval-Augmented Generation, Large Language Models, Prompting & Reasoning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Hailey Schoelkopf
- Sean Welleck
- Zhangir Azerbayev
- Xingyao Wang
- Hao Peng
- Keiran Paster
- Marco Dos Santos
- Albert Qiaochu Jiang
In Computational Theory and Mathematics
- Zhangir Azerbayev
- Hailey Schoelkopf
- Albert Qiaochu Jiang
- Heng Ji
- Hao Peng
- Sean Welleck
- Rahul Saha
- Keiran Paster
In Large Language Models
- Hailey Schoelkopf
- Sean Welleck
- Stella Biderman
- Heng Ji
- Hao Peng
- Albert Qiaochu Jiang
- Zhangir Azerbayev
- Rahul Saha
In Prompting & Reasoning
Citations in the corpus
- Llemma: An Open Language Model For Mathematics (2023, methodology): “On Lean 4 formal-to-formal proving, L LEMMA -7b improves upon its Code Llama initialization, and performs similar to ReProver (Yang et al., 2023), a retrieval-augmented language model finetuned for tactic prediction.”
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback (2023, background): “…focus on single-turn input-output (Hendrycks et al., 2020; Chen et al., 2021) and often overlook user-provided natural language feedback (Liu et al., 2023d; Deng et al., 2023b; Yang et al., 2023a; Shridhar et al., 2020), creating a discrepancy between real-world use cases and evaluation.”
- The rise and potential of large language model based agents: a survey (2025, unknown): “Currently, numerous efforts in various specialized domains aim to overcome this challenge [308–310].”
- AlphaEvolve: A coding agent for scientific and algorithmic discovery (2025, unknown): “…the FunSearch approach [24, 83] established LLM-guided evolution as a powerful tool for discovering witnesses for, and counterexamples to, mathematical statements—a problem that is complementary to that of finding formal and informal proofs of mathematical statements [3, 19, 98, 99, 110, 111].”
- LLMSTEP: LLM proofstep suggestions in Lean (2023, methodology): “We compare the Pythia model to ReProver [19] without retrieval.”
- CRANE: Reasoning with constrained LLM generation (2025, unknown): “…systems that interact with traditional software tools like Python interpreters (OpenAI, 2024; Chen et al., 2023) for code generation (Suresh et al., 2024a;b; Jiang et al., 2024), logical solvers (Pan et al., 2023; Olaus-* son et al., 2023), and theorem provers (Wu et al., 2022; Yang et al., 2023).”