CMATH: Can Your Language Model Pass Chinese Elementary School Math Test?

By Tianwen Wei, Jian Luan, Wei David Liu, Shuang Dong, B. Wang. 2023

84 citations recorded by the source corpus.

We present the Chinese Elementary School Math Word Problems (CMATH) dataset, comprising 1.7k elementary school-level math word problems with detailed annotations, source from actual Chinese workbooks and exams. This dataset aims to provide a benchmark tool for assessing the following question: to what grade level of elementary school math do the abilities of popular large language models (LLMs) correspond? We evaluate a variety of popular LLMs, including both commercial and open-source options, and discover that only GPT-4 achieves success (accuracy $\geq$ 60\%) across all six elementary school grades, while other models falter at different grade levels. Furthermore, we assess the robustness of several top-performing LLMs by augmenting the original problems in the CMATH dataset with distracting information. Our findings reveal that GPT-4 is able to maintains robustness, while other model fail. We anticipate that our study will expose limitations in LLMs' arithmetic and reasoning capabilities, and promote their ongoing development and advancement.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Maosong Sun
3. Xu Han
4. Shengding Hu
5. Rui Zhang
6. Peiyi Wang
7. Zhihong Shao
8. Jingren Zhou

In Computational Theory and Mathematics

1. Maosong Sun
2. Zhiyuan Liu

In Large Language Models

1. Zhiyuan Liu
2. Maosong Sun
3. Rui Zhang
4. Peiyi Wang
5. Zhihong Shao
6. Shengding Hu
7. Jingren Zhou
8. Renze Lou

In Evaluation & Benchmarks

1. Zhiyuan Liu
2. Maosong Sun
3. Xu Han
4. Rui Zhang
5. Peiyi Wang
6. Zhihong Shao
7. Shengding Hu
8. Jingren Zhou

Citations in the corpus

• DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (2024, methodology): “Chinese benchmarks include MGSM-zh (Shi et al., 2023), CMATH (Wei et al., 2023), Gaokao-MathCloze (Zhong et al., 2023), and Gaokao-MathQA (Zhong et al., 2023).”
• Tool learning with large language models: a survey (2024, unknown): “On the other hand, the adoption of the tool learning paradigm bolsters the robustness of responses and transparency of generation processes, thereby enhancing interpretability and user trust, as well as improving system robustness and adaptability.”
• The Lessons of Developing Process Reward Models in Mathematical Reasoning (2025, unknown): “Step Verification There are two primary approaches to evaluating the correctness of reasoning steps.”
• Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement (2024, methodology): “5B, outperforms all specific baselines on GSM8K, MATH, CMATH, GaoKao Math Cloze, and Gaokao Math QA.”
• Large Language Models for Mathematical Reasoning: Progresses and Challenges (2024, methodology): “Wei et al. (2023) assessed the robustness of several top-performing LLMs by augmenting the original problems in the curated CMATH dataset with distracting information.”
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems (2024, background): “Previous bench-1 marks (Koncel-Kedziorski et al., 2016; Wang et al., 2017; Ling et al., 2017; Amini et al., 2019; Cobbe et al., 2021; Wei et al., 2023) focus predominantly on math word problems (WMPs) which involve four basic arithmetic operations with single or multiple operation steps (Lu et al.,…”