ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

By Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu and 13 more. 2021

607 citations recorded by the source corpus.

Pre-trained models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks. Recent works such as T5 and GPT-3 have shown that scaling up pre-trained language models can improve their generalization abilities. Particularly, the GPT-3 model with 175 billion parameters shows its strong task-agnostic zero-shot/few-shot learning capabilities. Despite their success, these large-scale models are trained on plain texts without introducing knowledge such as linguistic knowledge and world knowledge. In addition, most large-scale models are trained in an auto-regressive way. As a result, this kind of traditional fine-tuning approach demonstrates relatively weak performance when solving downstream language understanding tasks. In order to solve the above problems, we propose a unified framework named ERNIE 3.0 for pre-training large-scale knowledge enhanced models. It fuses auto-regressive network and auto-encoding network, so that the trained model can be easily tailored for both natural language understanding and generation tasks with zero-shot learning, few-shot learning or fine-tuning. We trained the model with 10 billion parameters on a 4TB corpus consisting of plain texts and a large-scale knowledge graph. Empirical results show that the model outperforms the state-of-the-art models on 54 Chinese NLP tasks, and its English version achieves the first place on the SuperGLUE benchmark (July 3, 2021), surpassing the human performance by +0.8% (90.6% vs. 89.8%).

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Ethical and social risks of harm from Language Models (2021, background): “, 2019a) for language understanding (He et al., 2021; Sun et al., 2021; Wang et al., 2021).”
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, methodology): “The advantage of this method is that, given a suite of appropriate prompts, a single LM trained in an entirely unsupervised fashion can be used to solve a great number of tasks [9, 131].”
Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning (2021, background): “0 (2.3TB Chinese text data and 300GB English text data) [11], and ERNIE 3.0 (4TB) [21].”
Deep Bidirectional Language-Knowledge Graph Pretraining (2022, neutral): “Several existing works (Zhang et al. 2019; Xiong et al. 2020; Wang et al. 2021; Agarwal et al. 2021; Sun et al. 2021) propose methods for self-supervised pretraining, but they fuse text and KG in a shallow or uni-directional manner.”
Infinity∞: Scaling Bitwise AutoRegressive Modeling for High-Resolution Image Synthesis (2024, positive): “AutoRegressive models[14, 21, 21, 58, 64, 73], on the other hand, harness the scalability and generalizability of language models[2, 3, 16, 28, 60, 63, 65, 66, 72] by employing a visual tokenizer[50, 67, 78] to convert images into discrete tokens and causally optimize these to-kens, allowing image…”
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction (2024, positive): “The advent of GPT series [48, 49, 9, 45, 1] and other autoregressive (AR) large language models (LLMs) [13, 3, 27, 63, 64, 70, 60, 4, 61] has heralded a new epoch in the field of artificial intelligence.”
Large Language Models Are Reasoning Teachers (2022, background): “Recently, large language models (LLMs) have demonstrated in-context generalization capabilities: performing downstream tasks simply by conditioning on few in-context exemplars or plain natural language task descriptions (Brown et al., 2020; Sun et al., 2021).”
Humans or LLMs as the Judge? A Study on Judgement Bias (2024, background): “We test whether judges are distracted by formats.”
Pre-Trained Language Models for Text Generation: A Survey (2022, methodology): “Knowledge-Enriched PLMs. Recent work has found that integrating knowledge from external knowledge sources can enhance the text generation performance of PLMs [229, 285].”
A Comprehensive Overview of Large Language Models (2023, positive): “1.6 ERNIE 3.0 [95]: ERNIE 3.0 takes inspiration from multi-task learning to build a modular architecture using Transformer-XL [96] as the backbone.”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, positive): “This LLM also approached or exceeded state-of-the-art performance across MedMCQA [100], PubMedQA [101], and MMLU clinical topics datasets [102].”
Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “In the Application layer, LLMs and KGs can be integrated to address various real-world applications, such as search engines [100], recommender systems [10], and AI assistants [101].”
M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretraining (2021, background): “[19; 49; 47; 41] Sparse expert models with large model capacity are capable of fast training owing to the combination of data parallelism and expert parallelism [38; 17; 13; 35], and it is even accessible to train a 1-trillion-parameter transformer with no more than 500 GPUs [45].”
CValues: Measuring the Values of Chinese Large Language Models from Safety to Responsibility (2023, methodology): “This trend also inspires the rapid development of Chinese LLMs, such as PLUG with 27B parameters for language understanding and generation (ModelScope, 2021), Pangu-α with 200B parameters (Zeng et al., 2021) and ERNIE 3.0 with 260B parameters (Sun et al., 2021).”
Black-Box Tuning for Language-Model-as-a-Service (2022, methodology): “1 Therefore, large PTMs such as GPT-3 (Brown et al., 2020), ERNIE 3.0 (Sun et al., 2021) and Yuan 1.0 (Wu et al., 2021) are usually released as a service, allowing users to access these powerful models through black-box APIs.”