ERNIE 3.0 Titan: Exploring Larger-scale Knowledge Enhanced Pre-training for Language Understanding and Generation

By Shuohuan Wang, Yu Sun, Yang Xiang, Zhihua Wu, Siyu Ding, Weibao Gong, Shi Feng, Junyuan Shang, Yanbin Zhao, Chao Pang and 21 more. 2021

88 citations recorded by the source corpus.

Pre-trained language models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks. GPT-3 has shown that scaling up pre-trained language models can further exploit their enormous potential. A unified framework named ERNIE 3.0 was recently proposed for pre-training large-scale knowledge enhanced models and trained a model with 10 billion parameters. ERNIE 3.0 outperformed the state-of-the-art models on various NLP tasks. In order to explore the performance of scaling up ERNIE 3.0, we train a hundred-billion-parameter model called ERNIE 3.0 Titan with up to 260 billion parameters on the PaddlePaddle platform. Furthermore, we design a self-supervised adversarial loss and a controllable language modeling loss to make ERNIE 3.0 Titan generate credible and controllable texts. To reduce the computation overhead and carbon emission, we propose an online distillation framework for ERNIE 3.0 Titan, where the teacher model will teach students and train itself simultaneously. ERNIE 3.0 Titan is the largest Chinese dense pre-trained model so far. Empirical results show that the ERNIE 3.0 Titan outperforms the state-of-the-art models on 68 NLP datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

Citations in the corpus

Predictability and Surprise in Large Generative Models (2022, background): “0 Titan [70], Baidu, China; Gopher [56], DeepMind, USA/UK; FLAN [71] & LaMDA [68], Google, USA; Pan Gu [78] Huawei, China; Yuan 1.”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “We compare GLM-130B to the largest existing Chinese monolingual language model—the 260B ERNIE Titan 3.0 (Wang et al., 2021).”
Designing Effective Sparse Expert Models (2022, background): “Closed book WebQA achieves a 47.4 accuracy (prior best of 42.8 from Roberts et al. (2020) and exceeds results from the zero-shot performance of the ERNIE 3.0 Titan 260B dense parameter model (Wang et al., 2021)).”
Cerebras-GPT: Open Compute-Optimal Language Models Trained on the Cerebras Wafer-Scale Cluster (2023, methodology): “Further, when training models with a batch size smaller than the critical batch size, learning rate should be reduced linearly proportional to the reduction in batch size–consistent with the ﬁndings in (Shallue et al., 2018; Yang et al., 2021).”
Uncovering ChatGPT’s Capabilities in Recommender Systems (2023, methodology): “0 Titan [39] with 260B parameters, BLOOM [33] with 175B parameters, and PaLM [5] with 540B parameters.”
A Comprehensive Overview of Large Language Models (2023, unknown): “1.11 ERNIE 3.0 TITAN [102]: ERNIE 3.0 Titan extends ERNIE 3.0 by training a larger model with 26x the number of parameters of the latter.”
ST-MoE: Designing Stable and Transferable Sparse Expert Models (2022, background): “Closed book WebQA achieves a 47.4 accuracy (prior best of 42.8 from Roberts et al. (2020) and exceeds results from the zero-shot performance of the ERNIE 3.0 Titan 260B dense parameter model (Wang et al., 2021)).”
A Survey of Large Language Models (2023, unknown): “0 Titan [95] Dec-2021 260 - - - 300B tokens 2048 V100 28 d X GLaM [96] Dec-2021 1200 - - - 280B tokens - 1024 TPU v4 574 h X InstructGPT [61] Jan-2022 175 GPT-3 X X - X AlphaCode [97] Feb-2022 41 - - - 967B tokens Jul-2021 - - Chinchilla [34] Mar-2022 70 - - - 1.”