PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation

By Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, Kaisheng Wang, Xiaoda Zhang, Chen Li and 26 more. 2021

243 citations recorded by the source corpus.

Large-scale Pretrained Language Models (PLMs) have become the new paradigm for Natural Language Processing (NLP). PLMs with hundreds of billions parameters such as GPT-3 have demonstrated strong performances on natural language understanding and generation with \textit{few-shot in-context} learning. In this work, we present our practice on training large-scale autoregressive language models named PanGu-$\alpha$, with up to 200 billion parameters. PanGu-$\alpha$ is developed under the MindSpore and trained on a cluster of 2048 Ascend 910 AI processors. The training parallelism strategy is implemented based on MindSpore Auto-parallel, which composes five parallelism dimensions to scale the training task to 2048 processors efficiently, including data parallelism, op-level model parallelism, pipeline model parallelism, optimizer model parallelism and rematerialization. To enhance the generalization ability of PanGu-$\alpha$, we collect 1.1TB high-quality Chinese data from a wide range of domains to pretrain the model. We empirically test the generation ability of PanGu-$\alpha$ in various scenarios including text summarization, question answering, dialogue generation, etc. Moreover, we investigate the effect of model scales on the few-shot performances across a broad range of Chinese NLP tasks. The experimental results demonstrate the superior capabilities of PanGu-$\alpha$ in performing various tasks under few-shot or zero-shot settings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, background): “Other models have achieved strong results targeting specific non-English languages, such as Zeng et al. (2021) for Chinese.”
Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “Since then, several other hundred-billion scale dense transformer models have been announced, among them are Jurassic-1 [34], Yuan 1.0 [74], PanGu-α [78] and Gopher [53].”
Deduplicating Training Data Makes Language Models Better (2021, methodology): “MinHash (Broder, 1997) is an approximate matching algorithm widely used in large-scale deduplication tasks (Versley and Panchenko, 2012; Gabriel et al., 2018; Gyawali et al., 2020), including to deduplicate the training set for a large Chinese-language LM (Zeng et al., 2021).”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, background): “, 2022) and exploit (Shoeybi et al., 2019; Brown et al., 2020; Smith et al., 2022; Chowdhery et al., 2022; Rae et al., 2021; Wang et al., 2021; Zeng et al., 2021; Zhang et al., 2022) the benefits of scale.”
GPT-NeoX-20B: An Open-Source Autoregressive Language Model (2022, methodology): “In particular, systematic analysis has shown signifi- cant benefits in terms of reducing the leakage of training data (Lee et al., 2021; Zhang et al., 2021; Carlini et al., 2022; Kandpal et al., 2022).”
The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only (2023, methodology): “For fuzzy duplicates, methods based on locally-sensitive hashes such as MinHash (Broder, 1997) or SimHash (Charikar, 2002) have been adopted for the pretraining data of large language models (Brown et al., 2020; Zeng et al., 2021; Rae et al., 2021).”
Predictability and Surprise in Large Generative Models (2022, background): “0 Titan [70], Baidu, China; Gopher [56], DeepMind, USA/UK; FLAN [71] & LaMDA [68], Google, USA; Pan Gu [78] Huawei, China; Yuan 1.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, background): “Later, despite many 100B-scale LLMs (Lieber et al., 2021; Thoppilan et al., 2022; Rae et al., 2021; Smith et al., 2022; Chowdhery et al., 2022; Wu et al., 2021; Zeng et al., 2021; Wang et al., 2021) in both English and Chinese, they are not available to public or only accessible via limited APIs.”
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, background): “…makes up the backbone of the GPT series of models [Radford et al., 2018, 2019, Brown et al., 2020] as well as many other recent record-breaking LLMs [Zeng et al., 2021, Kim et al., 2021, Smith et al., 2022, Thoppilan et al., 2022, Rae et al., 2021, Hoffmann et al., 2022, Chowdhery et al., 2022].”
ZeroPrompt: Scaling Prompt-Based Pretraining to 1, 000 Tasks Improves Zero-Shot Generalization (2022, neutral): “We compare ZeroPrompt with state-of-the-art large-scale Chinese pretrained models, Pangu-α (13B decoder) (Zeng et al., 2021), CPM-2 (11B encoder-decoder) (Zhang et al., 2021), and the finetuned RoBERTa-large model (Liu et al., 2019).”
MAGMA - Multimodal Augmentation of Generative Models through Adapter-based Finetuning (2021, background): “While the performance of Frozen on VL benchmarks falls short compared to the state of the art, we feel the approach is promising due to its practicality, and the public availability of large, pretrained LMs such as GPT-J [54], PanGu-α [59], and GPT-Neo [3].”
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X (2023, background): “As shown in Figure 2, the input of the top query layer replaces the query input 𝑋 𝑖𝑛 by the query Table 1: Large pre-trained language models related to programming languages in the literature.”
Pre-Trained Models: Past, Present and Future (2021, background): “…boosted by the wide use of distributed computing devices and strategies, we can further advance the parameter scale of PTMs from million-level to billion-level (Brown et al., 2020; Lepikhin et al., 2021; Zeng et al., 2021; Zhang et al., 2020c, 2021a) and even trillion-level (Fedus et al., 2021).”
What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformers (2021, neutral): “Recently, a GPT-like language model trained on Chinese corpora is being actively re-searched concurrently with our work (Zeng et al., 2021).”
Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning (2021, methodology): “However, both GPT-3 and PanGu-α with singleton architecture, exhibits good performance on Zero-Shot and Few-Shot learning[6, 7].”
ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation (2021, methodology): “0 used by CPM-2 [20] (2.3TB Chinese data and 300GB English data) and PanGu Corpus [22] (1.1TB).”
CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Evaluations on HumanEval-X (2023, methodology): “We use an extra query layer (Zeng et al., 2021) on top of all other transformer layers to obtain the final embedding through attention.”