Evaluating Language Models for Mathematics through Interactions

By Katherine M. Collins, Albert Qiaochu Jiang, Simon Frieder, L. Wong, Miri Zilka, Umang Bhatt, Thomas Lukasiewicz, Yuhuai Wu, Josh Tenenbaum, William Hart and 3 more. 2023

114 citations recorded by the source corpus.

Significance Large language models (LLMs) are increasingly powerful, but their evaluation is often static, which does not consider LLMs’ performance when interacting with humans. We develop a platform to empower interactive evaluation, observe real mathematicians interacting with and evaluating LLMs in theorem proving to study how people solve problems with their assistance, and taxonomize the interactions to enable in-depth analyses. Our work systematically studies the interactive evaluation of LLMs in target settings and provides takeaways for people using and developing LLMs.

Subfields: Artificial Intelligence, Information Systems, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Prompting & Reasoning

Citations in the corpus

Llemma: An Open Language Model For Mathematics (2023, methodology): “Due to high computational costs of search, language models applied to this domain have traditionally been small, but recent work has demonstrated promise in the use of larger models (First et al., 2023; Jiang et al., 2023).”
MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models (2023, background): “…the rapid development of large language models (LLMs) which emerge as the favored approach for various applications and demonstrate multi-dimensional abilities, including instruction following [7, 38, 55, 67], coding assistance [8, 36, 44, 51], and mathematical problem-solving [14, 28, 43, 79].”
AlphaEvolve: A coding agent for scientific and algorithmic discovery (2025, unknown): “…the FunSearch approach [24, 83] established LLM-guided evolution as a powerful tool for discovering witnesses for, and counterexamples to, mathematical statements—a problem that is complementary to that of finding formal and informal proofs of mathematical statements [3, 19, 98, 99, 110, 111].”
OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems (2024, background): “The emergence of LLMs has facilitated notable advancements in the domain of natu-ral language proof (Jiang et al., 2023).”
OpenWebMath: An Open Dataset of High-Quality Mathematical Web Text (2023, methodology): “In contrast with prior works that extract text from Common Crawl such as C4 (Collins et al., 2023), The Pile (Gao et al., 2020), and RefinedWeb (Penedo et al., 2023), we chose to make a mostly custom pipeline for extracting the main content from HTML documents.”