CPM-2: Large-scale Cost-effective Pre-trained Language Models

By Zhengyan Zhang, Yuxian Gu, Xu Han, Shengqi Chen, Chaojun Xiao, Zhenbo Sun, Yuan Yao, Fanchao Qi, Jian Guan, Pei Ke and 9 more. 2021

101 citations recorded by the source corpus.

In recent years, the size of pre-trained language models (PLMs) has grown by leaps and bounds. However, efficiency issues of these large-scale PLMs limit their utilization in real-world scenarios. We present a suite of cost-effective techniques for the use of PLMs to deal with the efficiency issues of pre-training, fine-tuning, and inference. (1) We introduce knowledge inheritance to accelerate the pre-training process by exploiting existing PLMs instead of training models from scratch. (2) We explore the best practice of prompt tuning with large-scale PLMs. Compared with conventional fine-tuning, prompt tuning significantly reduces the number of task-specific parameters. (3) We implement a new inference toolkit, namely InfMoE, for using large-scale PLMs with limited computational resources. Based on our cost-effective pipeline, we pre-train two models: an encoder-decoder bilingual model with 11 billion parameters (CPM-2) and its corresponding MoE version with 198 billion parameters. In our experiments, we compare CPM-2 with mT5 on downstream tasks. Experimental results show that CPM-2 has excellent general language intelligence. Moreover, we validate the efficiency of InfMoE when conducting inference of large-scale models having tens of billions of parameters on a single GPU. All source code and model parameters are available at https://github.com/TsinghuaAI/CPM.

Subfields: Artificial Intelligence, Large Language Models, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Maosong Sun
3. Minlie Huang
4. Xu Han
5. Xiaoyan Zhu
6. Jie Tang
7. Yang Liu
8. Jianfeng Gao

In Large Language Models

1. Zhiyuan Liu
2. Haifeng Wang
3. Hua Wu
4. Jie Tang
5. Minlie Huang
6. Maosong Sun
7. Xu Han
8. Hongxia Yang

In Efficiency & Compression

1. Shi Feng
2. Chujie Zheng
3. Ajit Rajasekharan
4. Minlie Huang
5. Maosong Sun
6. Zhilin Yang
7. Jie Tang
8. Yuxian Gu

Citations in the corpus

• ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization (2022, methodology): “We compare ZeroPrompt with state-of-the-art large-scale Chinese pretrained models, Pangu-α (13B decoder) (Zeng et al., 2021), CPM-2 (11B encoder-decoder) (Zhang et al., 2021), and the finetuned RoBERTa-large model (Liu et al., 2019).”
• PPT: Pre-trained Prompt Tuning for Few-shot Learning (2021, methodology): “For Chinese datasets, we do PT based on a 11B model CPM-2.”
• ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, methodology): “0 with 10 previous SoTA baselines including CPM-2[20], ERNIE 2.”
• Prompttts: Controllable Text-To-Speech With Text Descriptions (2022, background): “Recent research has achieved great success in text and image generation guided with a text description as prompt [1, 2, 3, 4] (e.”
• EVA2.0: Investigating Open-domain Chinese Dialogue Systems with Large-scale Pre-training (2022, methodology): “Although similar model designs are also used in previous works, including both general pre-trained models [11, 18] and dialogue-specific pre-trained models [4–6], they do not include the discussions of some variants of the model that can have a non-trivial impact on the final performance.”
• A Comprehensive Overview of Large Language Models (2023, unknown): “1: The trends in the number of LLM models introduced over the years.”
• MSP: Multi-Stage Prompting for Making Pre-trained Language Models Better Translators (2021, methodology): “561 Zhang et al. (2021) investigate using prompt tuning 562 for steering CPM-2 model to the WMT20 English- 563 Chinese translation task.”
• AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, methodology): “KIPT [72], [73] which pretrain a model using both SSL and KD can be used.”
• A Survey of Large Language Models (2023, unknown): “For example, OPT [79], OPT-IML [83], BLOOM [66], and BLOOMZ [82] have nearly the same number of parameters as GPT-3 (175B version), while GLM [80] 5 2020 2023 2021 2022 1-4 5-8 9-10 11-12 1-3 4-6 7-10 11-12 T5 GPT-3 WebGPT OPTGLMBLOOMZ Galatica mT0 LLaMA AlphaCode 2019 FLAN Chinchilla InstructGPT GP”
• ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, methodology): “We compare ERNIE 3.0 Titan with ERNIE 3.0 and 10 previous SoTA baselines including CPM-2 [28], ERNIE 2.0 [52], SKEP [93], RoBERTa-wwm-ext-large [94] (marked as RoBERTa*), ALBERT [95], MacBERT [96], Zen 2.0 [97] and crossed BERT siamese BiGRU [98] (marked as BERT_BiGRU*).”
• Pre-Trained Language Models for Text Generation: A Survey (2022, unknown): “Enc-Dec LM MASS [168]; T5 [154]; BART [99]; ProphetNet [150]; CPM-2 [216]”
• M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining (2021, background): “[19; 49; 47; 41] Sparse expert models with large model capacity are capable of fast training owing to the combination of data parallelism and expert parallelism [38; 17; 13; 35], and it is even accessible to train a 1-trillion-parameter transformer with no more than 500 GPUs [45].”
• On Transferability of Prompt Tuning for Natural Language Understanding (2021, background): “PLMs can achieve remarkable natural language understanding (NLU) performance, and various large PLMs are continually developed (Raffel et al., 2020; Zhang et al., 2021; Zeng et al., 2021; Wei et al., 2021; Sun et al., 2021), which contain up to hundreds of billions of parameters.”