CLUE: A Chinese Language Understanding Evaluation Benchmark

By Xuanwei Zhang, Lu Li, Hai Hu, Chenjie Cao, Weitang Liu, Junyi Li, Yudong Li, Kai Sun, Yechen Xu, Yiming Cui and 20 more. 2020

460 citations recorded by the source corpus.

The advent of natural language understanding (NLU) benchmarks for English, such as GLUE and SuperGLUE allows new NLU models to be evaluated across a diverse set of tasks. These comprehensive benchmarks have facilitated a broad range of research and applications in natural language processing (NLP). The problem, however, is that most such benchmarks are limited to English, which has made it difficult to replicate many of the successes in English NLU for other languages. To help remedy this issue, we introduce the first large-scale Chinese Language Understanding Evaluation (CLUE) benchmark. CLUE is an open-ended, community-driven project that brings together 9 tasks spanning several well-established single-sentence/sentence-pair classification tasks, as well as machine reading comprehension, all on original Chinese text. To establish results on these tasks, we report scores using an exhaustive set of current state-of-the-art pre-trained Chinese models (9 in total). We also introduce a number of supplementary datasets and additional tools to help facilitate further progress on Chinese NLU. Our benchmark is released at https://www.cluebenchmarks.com

Subfields: Artificial Intelligence, General Social Sciences, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Minlie Huang
2. Zhiyuan Liu
3. Maosong Sun
4. Jianfeng Gao
5. Jie Tang
6. Mohit Bansal
7. Graham Neubig
8. Xu Han

In General Social Sciences

1. Haifeng Wang
2. Shi Feng
3. Xiaoyan Zhu
4. Ming Ding
5. Qun Liu
6. Anoop Kunchukuttan
7. Wenguang Chen
8. Minlie Huang

In Natural Language Processing

1. Maosong Sun
2. Zhiyuan Liu
3. Kyle Richardson
4. Minlie Huang
5. Xu Han
6. Qun Liu
7. Haifeng Wang
8. Hua Wu

In Evaluation & Benchmarks

1. Zhiyuan Liu
2. Maosong Sun
3. Xu Han
4. Mitesh M. Khapra
5. Shuohuan Wang
6. Tim Baldwin
7. W. Tam
8. Haifeng Wang

Citations in the corpus

• PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation (2021, methodology): “Text classification tasks, including TouTiao Text Classification for News Titles (TNEWS), IFLYTEK app description classification (IFLYTEK), Ant Financial Question Matching Corpus (AFQMC), and Chinese Scientific Literature (CSL) [39].”
• GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “Here we elaborate the prompts we use for CLUE (Xu et al., 2020) and FewCLUE (Xu et al.”
• C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models (2023, background): “…Chinese LLMs, such as GLM-130B (Zeng et al., 2023), Wenxin Yiyan (Baidu, 2023), and MOSS (OpenLMLab, 2023), the corresponding evaluation significantly lags behind, with the CLUE benchmark (Xu et al., 2020), the Chinese counterpart of GLUE (Wang et al., 2019), serving as the best available standard.”
• Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning (2021, methodology): “To our best knowledge, this is the largest Chinese text dataset compared with CLUECorpus2020 (100GB) [20], PanGu Corpus (1.1TB) [7], WuDaoCorpus2.”
• CPM: A Large-scale Generative Chinese Pre-trained Language Model (2020, methodology): “Although there are some previous works providing powerful Chinese pre-trained language models (Cui et al., 2020; Xu et al., 2020; Wei et al., 2019; Sun et al., 2019; Cui et al., 2019a), their capabilities are limited due to the model size.”
• DeepSeek LLM: Scaling Open-Source Language Models with Longtermism (2024, unknown): “We apply generation-based evaluation for TriviaQA, NaturalQuestions, DROP, MATH, GSM8K, HumanEval, MBPP, BBH, AGIEval, CLUEWSC, and CMath.”
• C-Pack: Packed Resources For General Chinese Embeddings (2023, background): “Particularly, the following labeled datasets are included, T 2 - Ranking [60], DuReader [20, 42], mMARCO [8], CMedQA-v2[65], multi-cpr[31], NLI-Zh 9 , cmnli[62] and ocnli[62].”
• Improving Sign Language Translation with Monolingual Data by Sign Back-Translation (2021, unknown): “They are all native signers from the deaf community and 4 of them are engaged in sign language educa-Table [51] tion.”
• C-Pack: Packaged Resources To Advance General Chinese Embedding (2023, background): “In the past few years, the community has put forward essential datasets to study Chinese text embeddings, such as CMNLI (Xu et al., 2020a), DuReader (He et al., 2017), T 2 Ranking (Xie et al., 2023).”
• A Survey of Large Language Models (2023, unknown): “Conditional Text Generation WMT’14,16,19,20,21,22 [271–276], Flores-101 [277], DiaBLa [278], CNN/DailyMail [279], XSum [280], WikiLingua [281], OpenDialKG [282] SuperGLUE [283], MMLU [284], BIG-bench Hard [285], CLUE [286]”
• ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, background): “While for TNEWS-FC, IFLYTEK-FC, CSLDCP, the negative sampling strategy is discarded in order to compare with Yuan 1.0.”
• DeepSeek-Coder-V2: Breaking the Barrier of Closed-Source Models in Code Intelligence (2024, unknown): “…BigBench Hard (BBH) (Suzgun et al., 2022), MMLU (Hendrycks et al., 2020), ARC (Clark et al., 2018), TriviaQA (Joshi et al., 2017), NaturalQuestions (Kwiatkowski et al., 2019), AGIEval (Zhong et al., 2023) ， CLUEWSC (Xu et al., 2020), C-Eval (Huang et al., 2023), and CMMLU (Li et al., 2023a).”
• ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2024, methodology): “The pruning ratios ranged from 0% to 97%, employing the strategy of deleting layers based on the importance proposed in this paper, resulting in models with 1 to 32 layers.”
• iNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages (2020, methodology): “, 2019), CLUE (Chinese) (Xu et al., 2020), and FLUE (French) (Le et al.”