CPM: A Large-scale Generative Chinese Pre-trained Language Model

By Zhengyan Zhang, Xu Han, Hao Zhou, Pei Ke, Yuxian Gu, Deming Ye, Yujia Qin, Yusheng Su, Haozhe Ji, Jian Guan and 15 more. 2020

132 citations recorded by the source corpus.

Pre-trained Language Models (PLMs) have proven to be beneficial for various downstream NLP tasks. Recently, GPT-3, with 175 billion parameters and 570GB training data, drew a lot of attention due to the capacity of few-shot (even zero-shot) learning. However, applying GPT-3 to address Chinese NLP tasks is still challenging, as the training corpus of GPT-3 is primarily English, and the parameters are not publicly available. In this technical report, we release the Chinese Pre-trained Language Model (CPM) with generative pre-training on large-scale Chinese training data. To the best of our knowledge, CPM, with 2.6 billion parameters and 100GB Chinese training data, is the largest Chinese pre-trained language model, which could facilitate several downstream Chinese NLP tasks, such as conversation, essay generation, cloze test, and language understanding. Extensive experiments demonstrate that CPM achieves strong performance on many NLP tasks in the settings of few-shot (even zero-shot) learning. The code and parameters are available at this https URL.

Subfields: Artificial Intelligence, General Social Sciences, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In General Social Sciences

In Large Language Models

Citations in the corpus

PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation (2021, neutral): “Table 9 compares PanGu-α 2.6B with CPM [3] 15, a recently released generative Chinese PLM with 2.6B parameters, on 16 downstream tasks in Chinese.”
ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization (2022, methodology): “Recent progress like GPT-3 (Brown et al., 2020) demonstrates the possibility of prompting on larger-scale models for zero-shot learning, but the performance of zero-shot generalization still falls short on many tasks compared to fully-supervised fine-tuning.”
Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning (2021, methodology): “• Mixture of Experts (MoE): Scaling the model size with Sparsely Gated Mixture-of-Experts (MoE), such as GShard[8], Switch Transformer[9], Wudao[10, 11] and M6[12] .”
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models (2021, background): “It is not surprising that the community is racing to break the record of the largest parameter counts [Radford et al., 2019, Brown et al., 2020, Dosovitskiy et al., 2020, Tolstikhin et al., 2021, Zhang et al., 2021, Naumov et al., 2019, Jumper et al., 2021].”
Controllable Generation from Pre-trained Language Models via Inverse Prompting (2021, methodology): “For generating baseline texts, for QA, we generate the prompting baseline using the base text generation code under the prompt format of ” 问题:${Question} 描述:${Description} 回答:”, for CPM we apply the same prompt format and use its recommended text generation code.”
WuDaoCorpora: A super large-scale Chinese corpora for pre-training language models (2021, background): “A large number of studies have consistently shown that training pre-trained language models (PLMs) on large corpora can help it learn general language representation (Devlin et al., 2019; Zhang et al., 2012; Liu et al., 1907; Lan et al., 1909; Yao et al., 1909; Sun et al., 1907; Yang et al., 2019).”
A Survey on Gender Bias in Natural Language Processing (2021, background): “In particular, still most of the recently published models do not include a study of (gender) bias and ethical considerations alongside their publication [Conneau et al. 2020; Devlin et al. 2019; Raffel et al. 2020; Zhang et al. 2020] with the noteworthy exclusion of GPT-3 [Brown et al.”
CPM-2: Large-scale Cost-effective Pre-trained Language Models (2021, methodology): “The comparisons between our models and CPM (Zhang et al., 2020) are presented in Table 1.”
EVA2.0: Investigating Open-domain Chinese Dialogue Systems with Large-scale Pre-training (2022, methodology): “The CPM family [18, 19] pioneers the Chinese pre-trained models.”
EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Training (2021, methodology): “Q&A is a special kind of interaction among users on the Internet.”
Pre-Trained Language Models for Text Generation: A Survey (2022, unknown): “CPM [277] and PanGu-α [265] practiced on training large-scale auto-regressive Chinese language models.”