Mathematical Capabilities of ChatGPT

By Simon Frieder, Luca Pinchetti, Ryan-Rhys Griffiths, Tommaso Salvatori, Thomas Lukasiewicz, Philipp Petersen, A. Chevalier, J. Berner. Neural Information Processing Systems, 2023

589 citations recorded by the source corpus.

We investigate the mathematical capabilities of two iterations of ChatGPT (released 9-January-2023 and 30-January-2023) and of GPT-4 by testing them on publicly available datasets, as well as hand-crafted ones, using a novel methodology. In contrast to formal mathematics, where large databases of formal proofs are available (e.g., the Lean Mathematical Library), current datasets of natural-language mathematics, used to benchmark language models, either cover only elementary mathematics or are very small. We address this by publicly releasing two new datasets: GHOSTS and miniGHOSTS. These are the first natural-language datasets curated by working researchers in mathematics that (1) aim to cover graduate-level mathematics, (2) provide a holistic overview of the mathematical capabilities of language models, and (3) distinguish multiple dimensions of mathematical reasoning. These datasets also test whether ChatGPT and GPT-4 can be helpful assistants to professional mathematicians by emulating use cases that arise in the daily professional activities of mathematicians. We benchmark the models on a range of fine-grained performance metrics. For advanced mathematics, this is the most detailed evaluation effort to date. We find that ChatGPT can be used most successfully as a mathematical assistant for querying facts, acting as a mathematical search engine and knowledge base interface. GPT-4 can additionally be used for undergraduate-level mathematics but fails on graduate-level difficulty. Contrary to many positive reports in the media about GPT-4 and ChatGPT's exam-solving abilities (a potential case of selection bias), their overall mathematical performance is well below the level of a graduate student. Hence, if your goal is to use ChatGPT to pass a graduate-level math exam, you would be better off copying from your average peer!

Subfields: Health Informatics, Information Systems, Safety Research

View Paper: https://www.semanticscholar.org/paper/3f2cb353c7528efafb847309ab1e1e95245740a4

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Health Informatics

In Information Systems

In Safety Research

Citations in the corpus

A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT (2023, background): “Other papers have looked at its mathematical reasoning abilities [35].”
SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (2023, background): “One percent of the speed of light is The momentum of the electron is given by The de Broglie wavelength of this electron is This wavelength is of atomic dimensions.”
A Survey on Evaluation of Large Language Models (2023, background): “, 2023), but ChatGPT still lacks mathematical reasoning (Bang et al., 2023; Frieder et al., 2023) abilities.”
A Brief Overview of ChatGPT: The History, Status Quo and Potential Future Development (2023, background): “Those factual errors may be due to errors and noise in the training data [100], [101].”
Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models (2023, methodology): “[7] constructed the GHOSTS natural language dataset, which consists of graduate-level math test questions.”
Large Language Models for Mathematical Reasoning: Progresses and Challenges (2024, methodology): “A n s w e r C ARP (Zhang et al., 2023a) 4.9K M Chinese GSM8K (Cobbe et al., 2021) 8.5K M Linguistically diverse MATH (Hendrycks et al., 2021) 12.5K H Problems are put into difficulty levels 1-5 PRM800K (Lightman et al., 2023) 12K H MATH w/ step-wise labels M ATH QA (Amini et al., 2019) 37K C GRE exa”
ChatGPT and Open-AI Models: A Preliminary Review (2023, methodology): “Mathematics ChatGPT’s mathematical skills are noticeably worse than those of a typical mathematics graduate student [63].”
Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models (2023, background): “In the field of mathematics, Frieder et al. [17] constructed the GHOSTS natural language dataset, which consists of graduate-level math test questions.”
OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems (2024, background): “As the complexity of the problems rises, some works (Hendrycks et al., 2021; Frieder et al., 2023; Arora et al., 2023) introduce competition-level problems integrating mathematical logic and background knowledge.”