WuDaoCorpora: A super large-scale Chinese corpora for pre-training language models

By Sha Yuan, Hanyu Zhao, Zhengxiao Du, Ming Ding, Xiao Liu, Yukuo Cen, Xu Zou, Zhilin Yang, Jie Tang. 2021

131 citations recorded by the source corpus.

Using large-scale training data to build a pre-trained language model (PLM) with a larger volume of parameters can significantly improve downstream tasks. For example, OpenAI trained the GPT3 model with 175 billion parameters on 570 GB English training data, enabling downstream applications building with only a small number of samples. However, there is a lack of Chinese corpus to support large-scale PLMs. This paper introduces a super large-scale Chinese corpora WuDaoCorpora, containing about 3 TB training data and 1.08 trillion Chinese characters. We also release the base version of WuDaoCorpora, containing about 200 GB training data and 72 billion Chinese characters. As a baseline, we train a model transformer-XL with 3 billion parameters on the base version to test the corpora ’ s effect. The results show that the models trained on this corpora can achieve excellent performance in Chinese. The data and model are available at https://data.wudaoai.cn and https://github.com/ THUDM/Chinese-Transformer-XL, respectively.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Minlie Huang
3. Maosong Sun
4. Jie Tang
5. Xu Han
6. Xiaoyan Zhu
7. Yang Liu
8. Mohit Bansal

In Large Language Models

1. Zhiyuan Liu
2. Minlie Huang
3. Jie Tang
4. Ming Ding
5. Xiao Liu
6. Maosong Sun
7. Xu Han
8. Niklas Muennighoff

In Natural Language Processing

1. Maosong Sun
2. Zhiyuan Liu
3. Xu Han
4. Minlie Huang
5. Yang Liu
6. Jie Tang
7. Peitian Zhang
8. Shitao Xiao

Citations in the corpus

• CogView: Mastering Text-to-Image Generation via Transformers (2021, methodology): “The dataset is an extension of project WudaoCorpora [52] 10 .”
• GLM-130B: An Open Bilingual Pre-trained Model (2022, background): “The pre-training data includes 1.2T Pile (train split) (Gao et al., 2020) English, 1.0T Chinese WudaoCorpora (Yuan et al., 2021), and 250G Chinese corpora (including online forums, encyclopedia, and QA) we crawl from the web, which form a balanced composition of English and Chinese contents.”
• The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, background): “Tashkeela ar Zerrouki and Balla (2017) TeCLa ca Carrino et al.”
• PPT: Pre-trained Prompt Tuning for Few-shot Learning (2021, methodology): “For prompt pre-training, we sample 10GB data from OpenWebText (Gokaslan et al., 2019) for English tasks and 10GB data from WuDaoCor-pora (Yuan et al., 2021) for Chinese tasks.”
• A Comprehensive Overview of Large Language Models (2023, unknown): “LLaMA CommonCrawl, C4 [11], Github, Wikipedia, Books, arXiv, StackExchange CR: BoolQ [197], PIQA [149], SIQA [209], HellaSwag [145], WinoGrande [154], ARC-Challenge [143], OpenBookQA [148]; QA: TriviaQA [153], Natural Questions [206]; RC: RACE-middle [198], RACE-high [198]; Mathematical Reasoning: M”
• C-Pack: Packaged Resources To Advance General Chinese Embedding (2023, methodology): “We create a massive training dataset of 100M text pairs, which integrates both labeled data and unlabeled data curated from Wudao (Yuan et al., 2021), one of the largest corpora for pre-training Chinese language models.”
• C-Pack: Packed Resources For General Chinese Embeddings (2023, unknown): “The most representative one is the Wudao corpus [63], which is the largest dataset of well-formatted articles for pre-training Chinese language models.”
• M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation (2024, unknown): “The data comprises Pile (Gao et al., 2020), Wudao (Yuan et al., 2021), and mC4 (Raffel et al., 2020) datasets.”
• CPM-2: Large-scale Cost-effective Pre-trained Language Models (2021, methodology): “We pre-train our model on WuDaoCorpus (Yuan et al., 2021), which contains 2.3TB cleaned Chinese data as well as 300GB cleaned English data.”