Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning

By Shaohua Wu, Xudong Zhao, Tong Yu, Rongguo Zhang, C. Shen, Hongli Liu, Feng Li, Hong Zhu, Jiangang Luo, Xuanwei Zhang. 2021

69 citations recorded by the source corpus.

Recent work like GPT-3 has demonstrated excellent performance of Zero-Shot and Few-Shot learning on many natural language processing (NLP) tasks by scaling up model size, dataset size and the amount of computation. However, training a model like GPT-3 requires huge amount of computational resources which makes it challengeable to researchers. In this work, we propose a method that incorporates large-scale distributed training performance into model architecture design. With this method, Yuan 1.0, the current largest singleton language model with 245B parameters, achieves excellent performance on thousands GPUs during training, and the state-of-the-art results on NLP tasks. A data processing method is designed to efficiently filter massive amount of raw data. The current largest high-quality Chinese corpus with 5TB high quality texts is built based on this method. In addition, a calibration and label expansion method is proposed to improve the Zero-Shot and Few-Shot performance, and steady improvement is observed on the accuracy of various tasks. Yuan 1.0 presents strong capacity of natural language generation, and the generated articles are difficult to distinguish from the human-written ones.

Subfields: Artificial Intelligence, Large Language Models, Prompting & Reasoning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Prompting & Reasoning

Citations in the corpus

Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model (2022, background): “Since then, several other hundred-billion scale dense transformer models have been announced, among them are Jurassic-1 [34], Yuan 1.0 [74], PanGu-α [78] and Gopher [53].”
GLM-130B: An Open Bilingual Pre-trained Model (2022, background): “Later, despite many 100B-scale LLMs (Lieber et al., 2021; Thoppilan et al., 2022; Rae et al., 2021; Smith et al., 2022; Chowdhery et al., 2022; Wu et al., 2021; Zeng et al., 2021; Wang et al., 2021) in both English and Chinese, they are not available to public or only accessible via limited APIs.”
What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization? (2022, methodology): “Sometimes called a preﬁx language model, this approach was introduced by [Liu et al., 2018] and was later explored as an architectural variant by [Raffel et al., 2020, Wu et al., 2021].”
What Language Model to Train if You Have One Million GPU Hours? (2022, background): “The majority of 100B+ language models have been trained in English, with notable exceptions in Chinese (Zeng et al., 2021; Wu et al., 2021) and Korean (Kim et al., 2021) models.”
EVA2.0: Investigating Open-domain Chinese Dialogue Systems with Large-scale Pre-training (2022, background): “PanGu-α [20] and Yuan 1.0 [21] boost the power of Chinese language models by pushing the model sizes to 200B and 245B. Mengzi [22] instead builds a lightweight yet still powerful Chinese model and is computation-friendly for deployment.”
REPLUG: Retrieval-Augmented Black-Box Language Models (2023, background): “Black-box Language Models Large language models (i.e., >100B), such as GPT-3 (Brown et al., 2020a), Codex (Chen et al., 2021a), and Yuan 1.0 (Wu et al., 2021), are not open-sourced due to commercial considerations and are only available as black-box APIs, through which users can send queries and…”
Black-Box Tuning for Language-Model-as-a-Service (2022, background): “Scaling pre-trained language models (PTMs) has shown increasing power on a wide range of NLP tasks (Devlin et al., 2019; Raffel et al., 2020; Brown et al., 2020; Fedus et al., 2021; Zhang et al., 2020; 2021b; Zeng et al., 2021; Sun et al., 2021; Qiu et al., 2020).”
A Comprehensive Overview of Large Language Models (2023, unknown): “Training Parallelism 3D parallelism, a combination of data, pipeline and tensor parallelism, is the most utilized training parallelism approach in LLMs [84], [14], [10], [9], [21], [96], [93].”
A Survey of Large Language Models (2023, unknown): “0 [93] Oct-2021 245 - - - 180B tokens 2128 GPU - X Anthropic [94] Dec-2021 52 - - - 400B tokens - X WebGPT [70] Dec-2021 175 GPT-3 - X - X Gopher [59] Dec-2021 280 - - - 300B tokens - 4096 TPU v3 920 h X ERNIE 3.”