FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark

By Xiaojing Lu, Chenyang Yuan, Xuanwei Zhang, Huining Yuan, Huilin Xu, Guoao Wei, Xiang Pan, Hai Hu. 2021

59 citations recorded by the source corpus.

Pretrained Language Models (PLMs) have achieved tremendous success in natural language understanding tasks. While different learning schemes -- fine-tuning, zero-shot, and few-shot learning -- have been widely explored and compared for languages such as English, there is comparatively little work in Chinese to fairly and comprehensively evaluate and compare these methods and thus hinders cumulative progress. In this paper, we introduce the Chinese Few-shot Learning Evaluation Benchmark (FewCLUE), the first comprehensive few-shot evaluation benchmark in Chinese. It includes nine tasks, ranging from single-sentence and sentence-pair classification tasks to machine reading comprehension tasks. We systematically evaluate five state-of-the-art (SOTA) few-shot learning methods (including PET, ADAPET, LM-BFF, P-tuning and EFL), and compare their performance with fine-tuning and zero-shot learning schemes on the newly constructed FewCLUE benchmark. Experimental results reveal that: 1) The effect of different few-shot learning methods is sensitive to the pre-trained model to which the methods are applied; 2) PET and P-tuning achieve the best overall performance with RoBERTa and ERNIE respectively. Our benchmark is used in the few-shot learning contest of NLPCC 2021. In addition, we provide a user-friendly toolkit, as well as an online leaderboard to help facilitate further progress on Chinese few-shot learning. We provide a baseline performance on different learning methods, a reference for future research.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “To date, we have finished testing on part of the two benchmarks, including 7 CLUE and 5 FewCLUE datasets (Cf. Appendix C.7 for details).”
P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks (2021, methodology): “For OntoNotes 5.0, we use the same train, develop, test split as (Xu et al., 2021).”
Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning (2021, methodology): “Yuan LM-13B and Yuan PLM-13B are evaluated on FewCLUE[23] and ZeroCLUE[20] (Table 7).”
mGPT: Few-Shot Learners Go Multilingual (2022, unknown): “as well as the adversarial benchmarking (Xu et al., 2021).”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “Moreover, benchmarks like FewGLUE [277], FLEX [276], and FewCLUE [275] which evaluate fewshot learning techniques should be extended to other”
ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, background): “• Text Classification (TC): TNEWS 14, TNEWS-FC [68], IFLYTEK [76], IFLYTEK-FC [68], THUCNEWS 15, CNSE [77], CNSS [77], CSLDCP [68].”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, background): “Finally, many benchmarks have been proposed for comparing few-shot approaches in a standardized way (e.g., Mishra et al., 2021; Bragg et al., 2021; Xu et al., 2021; Ye et al., 2021; Alex et al., 2021).”
A Comprehensive Overview of Large Language Models (2023, unknown): “Type Datasets/Benchmarks Multi-Task MMLU [242], SuperGLUE [3], BIG-bench [243], GLUE [244], BBH [243], CUGE [245], ZeroCLUE [246], FewCLUE [247], Blended Skill Talk [248], HELM [249], KLUE-STS [250] Language Understanding CoQA [251], WiC [252], Wikitext103 [253], PG19 [254], LCQMC [255], QQP [256], ”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, methodology): “Recently, FewCLUE [27] has become the oicial benchmark in the Chinese language, including IFLYTEK (long paper classiication ), CLUEWSC (pronoun disambiguation), CSL (paper keyword recognition), CSLDCP and other tasks.”