C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models

By Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Jiayi Lei, Fanchao Qi and 3 more. 2023

890 citations recorded by the source corpus.

New NLP benchmarks are urgently needed to align with the rapid development of large language models (LLMs). We present C-Eval, the first comprehensive Chinese evaluation suite designed to assess advanced knowledge and reasoning abilities of foundation models in a Chinese context. C-Eval comprises multiple-choice questions across four difficulty levels: middle school, high school, college, and professional. The questions span 52 diverse disciplines, ranging from humanities to science and engineering. C-Eval is accompanied by C-Eval Hard, a subset of very challenging subjects in C-Eval that requires advanced reasoning abilities to solve. We conduct a comprehensive evaluation of the most advanced LLMs on C-Eval, including both English- and Chinese-oriented models. Results indicate that only GPT-4 could achieve an average accuracy of over 60%, suggesting that there is still significant room for improvement for current LLMs. We anticipate C-Eval will help analyze important strengths and shortcomings of foundation models, and foster their development and growth for Chinese users.

Subfields: Artificial Intelligence, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Maosong Sun
3. Jianfeng Gao
4. Minlie Huang
5. Mohit Bansal
6. Jie Zhou
7. Hannaneh Hajishirzi
8. Xu Han

In Large Language Models

1. Zhiyuan Liu
2. Qun Liu
3. Xin Jiang
4. Maosong Sun
5. Lifeng Shang
6. Ji-rong Wen
7. Wayne Xin Zhao
8. Ziqing Yang

In Evaluation & Benchmarks

1. Zhiyuan Liu
2. Maosong Sun
3. Xu Han
4. Tim Baldwin
5. Bo Li
6. Fei Huang
7. Chang Zhou
8. Peiyi Wang

Citations in the corpus

• CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, background): “For exam-4 ple, SciBench (Wang et al., 2023a), an evaluation benchmark for complex scientific problems, and C-Eval (Huang et al., 2023), an evaluation benchmark for multilevel and multi-discipline Chinese, strive to gather data that is difficult to extract or convert into text to mitigate this…”
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, neutral): “We include several metrics for evaluating LLMs, including multiple-choice tasks such as MMLU (Hendrycks et al., 2021), CMMLU (Li et al., 2023a), and C-Eval (Huang et al., 2023), as well as calculating text similarity scores like BLEU-4 (Papineni et al., 2002) and ROUGE (Lin, 2004).”
• DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence (2024, positive): “…BigBench Hard (BBH) (Suzgun et al., 2022), MMLU (Hendrycks et al., 2020), ARC (Clark et al., 2018), TriviaQA (Joshi et al., 2017), NaturalQuestions (Kwiatkowski et al., 2019), AGIEval (Zhong et al., 2023) ， CLUEWSC (Xu et al., 2020), C-Eval (Huang et al., 2023), and CMMLU (Li et al., 2023a).”
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024, background): “Included benchmarks are categorized and listed as follows, where underlined benchmarks are in Chinese: Multi-subject multiple-choice datasets include MMLU (Hendrycks et al., 2020), C-Eval (Huang et al., 2023), and CMMLU (Li et al., 2023).”
• OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems (2024, background): “In MMLU-STEM (Hendrycks et al., 2020) and C-Eval-STEM (Huang et al., 2023), physics questions also adopt a multiple-choice format.”
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “…in a paradigm shift in nat-ural language processing (NLP), achieving unprecedented progress in language understanding (Hendrycks et al., 2021; Huang et al., 2023c), generation (Zhang et al., 2023f; Zhu et al., 2023b) and reasoning (Wei et al., 2022; Kojima et al., 2022; Qiao et al., 2022; Yu…”
• Hallucination of Multimodal Large Language Models: A Survey (2024, neutral): “Recently, the emergence of large language models (LLMs) [29, 81, 85, 99, 132] has dominated a wide range of tasks in natural language processing (NLP), achieving unprecedented progress in language understanding [39, 47], generation [128, 140] and reasoning [20, 58, 87, 107, 115].”
• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, positive): “…assessments in general knowledge, linguistic understanding, reasoning, mathematics, and coding tasks, such as MMLU [46], CMMLU [63], C-Eval [48], GAOKAO-Bench [149], TriviaQA [52], NaturalQuestions [58, 110], RACE [59], WinoGrande [103], HellaSwag [142], BigBench Hard [112], GSM8K-Test…”
• Baichuan 2: Open Large-scale Language Models (2023, methodology): “On general benchmarks like MMLU (Hendrycks et al., 2021a), CMMLU (Li et al., 2023), and C-Eval (Huang et al., 2023), Baichuan 2-7B achieves nearly 30% higher performance compared to Baichuan 1-7B. Specifically, Baichuan 2 is optimized to improve performance on math and code problems.”
• MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts (2023, background): “…math, medicine, coding and science (Bubeck et al., 2023; Nori et al., 2023; Chen et al., 2021; Fu et al., 2023; Sun et al., 2023; Wang et al., 2023; Huang et al., 2023, 2022; Liu et al., 2023b On the vision-language side, there are several generative foundation models such as LLaVA, miniGPT4,…”
• SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Models (2023, methodology): “C-EVAL [21] focuses on evaluating LLMs in a Chinese context, offering questions from humanities to science and engineering.”
• Large Language Diffusion Models (2025, neutral): “Chinese understanding: CMMLU [123] and C-Eval [124].”
• Qwen Technical Report (2023, methodology): “In our self-constructed evaluation dataset, the instructions are either manually written data or manual revised from public datasets, such as CLiB6, C-Eval (Huang et al., 2023), FacTool (Chern et al.”
• CMMLU: Measuring massive multitask language understanding in Chinese (2023, background): “Huang et al. (2023) released C-Eval, a benchmark that incorporates questions across four difficulty levels of elementary school, middle school, high school and college.”
• Humans or LLMs as the Judge? A Study on Judgement Bias (2024, background): “Although close-ended benchmarks such as MMLU (Hendrycks et al., 2020), C-Eval (Huang et al., 2023) are convenient to evaluate on, they often suffer from data contamination issue.”