Measuring Massive Multitask Chinese Understanding
By Hui Zeng. 2023
37 citations recorded by the source corpus.
The development of large-scale Chinese language models is flourishing, yet there is a lack of corresponding capability assessments. Therefore, we propose a test to measure the multitask accuracy of large Chinese language models. This test encompasses four major domains, including medicine, law, psychology, and education, with 15 subtasks in medicine and 8 subtasks in education. We found that the best-performing models in the zero-shot setting outperformed the worst-performing models by nearly 18.6 percentage points on average. Across the four major domains, the highest average zero-shot accuracy of all models is 0.512. In the subdomains, only the GPT-3.5-turbo model achieved a zero-shot accuracy of 0.693 in clinical medicine, which was the highest accuracy among all models across all subtasks. All models performed poorly in the legal domain, with the highest zero-shot accuracy reaching only 0.239. By comprehensively evaluating the breadth and depth of knowledge across multiple disciplines, this test can more accurately identify the shortcomings of the models.
Subfields: Natural Language Processing, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Natural Language Processing
In Evaluation & Benchmarks
Citations in the corpus
- C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models (2023, neutral): “The MMCU benchmark (Zeng, 2023) consists of tests from four major domains including medicine, law, psychology and education, which are also collected from Chinese College Entrance Exam, qualification test as well as university examinations.”
- CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility (2023, background): “Most of these focus on assessing the helpfulness of the LLMs, such as AGIEval (Zhong et al., 2023b) and MMCU (Zeng, 2023) for Chinese and English College Entrance Exams, M3KE (Liu et al., 2023) for knowledge evaluation on multiple major levels of Chinese education system and C-EVAL (Huang et al., 20”