CMMLU: Measuring massive multitask language understanding in Chinese

By Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, Tim Baldwin. Annual Meeting of the Association for Computational Linguistics, 2023

512 citations recorded by the source corpus.

As the capabilities of large language models (LLMs) continue to advance, evaluating their performance becomes increasingly crucial and challenging. This paper aims to bridge this gap by introducing CMMLU, a comprehensive Chinese benchmark that covers various subjects, including natural science, social sciences, engineering, and humanities. We conduct a thorough evaluation of 18 advanced multilingual- and Chinese-oriented LLMs, assessing their performance across different subjects and settings. The results reveal that most existing LLMs struggle to achieve an average accuracy of 50%, even when provided with in-context examples and chain-of-thought prompts, whereas the random baseline stands at 25%. This highlights significant room for improvement in LLMs. Additionally, we conduct extensive experiments to identify factors impacting the models' performance and propose directions for enhancing LLMs. CMMLU fills the gap in evaluating the knowledge and reasoning capabilities of large language models within the Chinese context.

Subfields: Artificial Intelligence, General Social Sciences

View Paper: https://www.semanticscholar.org/paper/bb9a44c94a89dbe00f0061d05c70a45064ff6ea6

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Qun Liu
2. Xin Jiang
3. Lifeng Shang
4. Ronan Le Bras
5. Bill Yuchen Lin
6. Damai Dai
7. Zhihong Shao
8. Wei Cheng

In General Social Sciences

1. Qun Liu
2. Xin Jiang
3. Daya Guo
4. Abhilasha Ravichander
5. Xianpei Han
6. Fei Mi
7. Cunxiang Wang
8. Ronan Le Bras

Citations in the corpus

• InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models (2025, unknown): “…cover comprehensive assessments in general knowledge, linguistic understanding, reasoning, mathematics, and coding tasks, such as MMLU [46], CMMLU [63], C-Eval [48], GAOKAO-Bench [149], TriviaQA [52], NaturalQuestions [58, 110], RACE [59], WinoGrande [103], HellaSwag [142], BigBench Hard [112],…”
• DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence (2024, unknown): “…BigBench Hard (BBH) (Suzgun et al., 2022), MMLU (Hendrycks et al., 2020), ARC (Clark et al., 2018), TriviaQA (Joshi et al., 2017), NaturalQuestions (Kwiatkowski et al., 2019), AGIEval (Zhong et al., 2023) ， CLUEWSC (Xu et al., 2020), C-Eval (Huang et al., 2023), and CMMLU (Li et al., 2023a).”
• Baichuan 2: Open Large-scale Language Models (2023, methodology): “, 2021a), CMMLU (Li et al., 2023), and C-Eval (Huang et al.”
• Yi: Open Foundation Models by 01.AI (2024, background): “Our prompt and post-processing strategy remains consistent with the default settings of the original benchmarks[2, 4, 7, 8, 10–12, 27, 28, 42, 50, 61– 63, 72, 74, 75, 89, 90].”
• Style Over Substance: Evaluation Biases for Large Language Models (2023, unknown): “Several recent studies suggest using a diverse set of NLP benchmarks for LLM evaluation (Hendrycks et al., 2021; Liang et al., 2022; Li et al., 2023b).”
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, unknown): “We include several metrics for evaluating LLMs, including multiple-choice tasks such as MMLU (Hendrycks et al., 2021), CMMLU (Li et al., 2023a), and C-Eval (Huang et al., 2023), as well as calculating text similarity scores like BLEU-4 (Papineni et al., 2002) and ROUGE (Lin, 2004).”
• A Survey on Evaluation of Large Language Models (2023, background): “[101] introduces CMMLU as a comprehensive Chinese proficiency standard and evaluates the performance of 18 LLMs across various academic disciplines.”
• Large Language Diffusion Models (2025, unknown): “Chinese understanding: CMMLU [123] and C-Eval [124].”
• Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention (2025, unknown): “By integrating hierarchical token compression with block selection within a trainable architecture, we achieve faster training and inference while matching or surpassing Full Attention performance.”
• InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency (2025, unknown): “MMLU-Pro [61], GAOKAO [177], IFEval [185]; (4) Agentic Tasks : SGP-Bench [102], ScreenSpot [16], ScreenSpot-v2 [150], OSWorld-G [156], VSI-Bench [161], ERQA [121], SpaCE-10 [38].”
• Aligning Large Language Models with Human: A Survey (2023, methodology): “Li et al. (2023b) follow the Post-answering strategy to construct instruction data for 52 popular languages using Google-Translate, then use these data to fine-tune LLaMA using the LoRA technology.”
• CodeScope: An Execution-based Multilingual Multitask Multidimensional Benchmark for Evaluating LLMs on Code Understanding and Generation (2023, background): “On the one hand, most studies tend to ignore the risk of data leakage, such as AGIEval (Zhong et al., 2023), a recent high-profile bilingual standardized test evaluation benchmark, the multilingual, multimodal and multilevel evaluation benchmark M3Exam (Zhang et al., 2023), and the interdisciplinary”