CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Model

By Xuanwei Zhang, Qianqian Dong. 2020

81 citations recorded by the source corpus.

In this paper, we introduce the Chinese corpus from CLUE organization, CLUECorpus2020, a large-scale corpus that can be used directly for self-supervised learning such as pre-training of a language model, or language generation. It has 100G raw corpus with 35 billion Chinese characters, which is retrieved from Common Crawl. To better understand this corpus, we conduct language understanding experiments on both small and large scale, and results show that the models trained on this corpus can achieve excellent performance on Chinese. We release a new Chinese vocabulary with a size of 8K, which is only one-third of the vocabulary size used in Chinese Bert released by Google. It saves computational cost and memory while works as good as original vocabulary. We also release both large and tiny versions of the pre-trained model on this corpus. The former achieves the state-of-the-art result, and the latter retains most precision while accelerating training and prediction speed for eight times compared to Bert-base. To facilitate future work on self-supervised learning on Chinese, we release our dataset, new vocabulary, codes, and pre-trained models on Github.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation (2021, methodology): “To the best of our knowledge, there are three Chinese text corpora that are above 100GB: (a) CLUECorpus2020 (100GB), which is retrieved from the Common Crawl dataset [15]; (b) the Chinese multi-modal pretraining data, released by [16] which contains 300GB texts; and (c) WuDaoCorpus6, which opens abo”
CLUE: A Chinese Language Understanding Evaluation Benchmark (2020, methodology): “CLUECorpus2020 (Xu et al., 2020) It contains 100 GB Chinese raw corpus, which is retrieved from Common Crawl.”
Multimodal datasets: misogyny, pornography, and malignant stereotypes (2021, methodology): “The sheer scale of this dataset has an enduring allure in the AI community and has been used as a seeding dataset in training pipelines of high-proﬁle projects 5 such as GPT-3 [34], CLUECorpus2020 [35], and XLM-R [36].”
M6: A Chinese Multimodal Pretrainer (2021, methodology): “[45] released a 100GB corpus named CLUECorpus2020, which is retrieved from the multilingual Common Crawl dataset.”
WuDaoCorpora: A super large-scale Chinese corpora for pre-training language models (2021, background): “However, the largest Chinese corpus previously is the CLUECorpus2020 [15] with only 100 GB data, which suggests it is difficult to satisfy the demand of training high-quality model for Chinese NLP tasks.”
AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “Monolingual (Chinese) CLUECorpus2020 [94] Collection of 100GB of Chinese common crawl text.”