ChID: A Large-scale Chinese IDiom Dataset for Cloze Test

By Chujie Zheng, Minlie Huang, Aixin Sun. 2019

108 citations recorded by the source corpus.

Cloze-style reading comprehension in Chinese is still limited due to the lack of various corpora. In this paper we propose a large-scale Chinese cloze test dataset ChID, which studies the comprehension of idiom, a unique language phenomenon in Chinese. In this corpus, the idioms in a passage are replaced by blank symbols and the correct answer needs to be chosen from well-designed candidate idioms. We carefully study how the design of candidate idioms and the representation of idioms affect the performance of state-of-the-art models. Results show that the machine accuracy is substantially worse than that of human, indicating a large space for further research.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Minlie Huang
3. Maosong Sun
4. Xiaoyan Zhu
5. Ting-Wei Liu
6. Xu Han
7. Jie Tang
8. Wanxiang Che

In Natural Language Processing

1. Zhiyuan Liu
2. Maosong Sun
3. Ting-Wei Liu
4. Yiming Cui
5. Guoping Hu
6. Shijin Wang
7. Wanxiang Che
8. Dian Yu

In Evaluation & Benchmarks

1. Zhiyuan Liu
2. Maosong Sun
3. Xu Han
4. Peiyi Wang
5. Zhihong Shao
6. Xiaoyan Zhu
7. Qun Liu
8. Minlie Huang

Citations in the corpus

• PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation (2021, methodology): “Cloze and completion tasks, including WPLC, CHID [30], PD&CFT [31], CMRC2017 [32], and CMRC2019 [33].”
• CLUE: A Chinese Language Understanding Evaluation Benchmark (2020, background): “ChID ChID (Zheng et al., 2019) is a large-scale Chinese IDiom cloze test dataset, which contains about 498,611 passages with 623,377 blanks covered from news, novels, and essays.”
• Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning (2021, background): “Reading Comprehension includes Chid[22] and Csl. Chid is a Chinese Idiom cloze test dataset.”
• CPM: A Large-scale Generative Chinese Pre-trained Language Model (2020, methodology): “Dataset: We use the Chinese IDiom cloze test dataset (ChID) (Zheng et al., 2019) as our benchmark dataset.”
• A Sentence Cloze Dataset for Chinese Machine Reading Comprehension (2020, background): “n MRC, Cui et al. (2019b) proposed CMRC 2018 dataset for Simplified Chinese, and Shao et al. (2018)proposed DRCDdatasetforTraditionalChinese, similartothepopulardataset SQuAD (Rajpurkar et al., 2016). Zheng et al. (2019) proposed a large-scale Chinese idiom cloze dataset. Though various efforts have ”
• FewCLUE: A Chinese Few-shot Learning Evaluation Benchmark (2021, background): “ChID ChID (Zheng et al., 2019) is a large-scale Chinese IDiom cloze test dataset, which covered from news, novels, and essays.”
• A Comprehensive Overview of Large Language Models (2023, unknown): “0 controllable dataset NLU: NLPCC2014-SC, SE-ABSA16_PHNS, SE-ABSA16_CAME, BDCI2019, EPRSTMT [293], COTE-BD [268], COTE-MFW [268], OCNLI [269], CMNLI [269], OCNLI-FC [293], CLUEWSC [269] CLUEWSC-FC [293], FinRE [270], SanWen [271], AFQMC [269], LCQMC [242], PAWS-X [231], BQ Corpus [272], CSL [269] CS”
• ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, unknown): “• For reading comprehension (CHID-FC) and winograd schema challenge task, we utilize pattern exploiting training [99] to reformulate such tasks cloze-style questions and perform gradient-based tuning.”
• DeepSeek LLM: Scaling Open-Source Language Models with Longtermism (2024, background): “However, tasks like CHID, which involve evaluating the usage of Chinese idioms, require the model to consume a significant number of Chinese tokens during pre-training.”
• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model (2024, background): “Chinese understanding and culture datasets include CHID (Zheng et al., 2019) and CCPM (Li et al., 2021).”
• ShortGPT: Layers in Large Language Models are More Redundant Than You Expect (2024, methodology): “…that the performance of the model pruned by our method significantly surpasses that of the baseline method, maintaining most of the large language model’s capabilities, such as reasoning, language understanding, knowledge retention, and examination performance, to the greatest extent possible.”