ERNIE 2.0: A Continual Pre-training Framework for Language Understanding

By Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, Haifeng Wang. 2019

902 citations recorded by the source corpus.

Recently pre-trained models have achieved state-of-the-art results in various language understanding tasks. Current pre-training procedures usually focus on training the model with several simple tasks to grasp the co-occurrence of words or sentences. However, besides co-occurring information, there exists other valuable lexical, syntactic and semantic information in training corpora, such as named entities, semantic closeness and discourse relations. In order to extract the lexical, syntactic and semantic information from training corpora, we propose a continual pre-training framework named ERNIE 2.0 which incrementally builds pre-training tasks and then learn pre-trained models on these constructed tasks via continual multi-task learning. Based on this framework, we construct several tasks and train the ERNIE 2.0 model to capture lexical, syntactic and semantic aspects of information in the training data. Experimental results demonstrate that ERNIE 2.0 model outperforms BERT and XLNet on 16 tasks including English tasks on GLUE benchmarks and several similar tasks in Chinese. The source codes and pre-trained models have been released at https://github.com/PaddlePaddle/ERNIE.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Natural Language Processing

Citations in the corpus

Adversarial Training for Large Neural Language Models (2020, methodology): “Inspired by ERNIE (Sun et al., 2019) and other continual pre-training approaches, we instead adopt a curriculum learning approach: ﬁrst train the model using the standard objective (1); and then continue the training with virtual adversarial training (3).”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “We initialize the text stream with the parameters from ERNIE 2.0 model [33], and train ERINE-VL with a total batch size of 512 for at least 500k steps on 8 V100 GPUs .”
Editing Factual Knowledge in Language Models (2021, methodology): “…2019) or TriviaQA (Joshi et al., 2017) (Lewis et al., 2020) with a standard seq2seq objective, i.e. , maximizing the model likelihood given the observed output sequence (Sutskever et al., 2011, 2014) and regularized with dropout (Srivastava et al., 2014) and label smoothing (Szegedy et al., 2016).”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “The third is continual pre-training (CPT) or continual meta-training.”
Pre-Trained Models: Past, Present and Future (2021, methodology): “ERNIE-M (Ouyang et al., 2020) proposes back-translation masked language modeling (BTMLM), and expands the scale of parallel corpora through back-translation mechanisms.”
Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation (2019, background): “Recent improvements in language understanding have been obtained through language model pre-training of expensive deep models on unlabeled text (Devlin et al., 2018; Yang et al., 2019b; Sun et al., 2019; Liu et al., 2019).”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, background): “A number of studies have been progressively advancing pretraining by improving pretraining tasks and designing more sophisticated model architectures [25, 26, 27, 28, 29, 30, 31].”
Pre-trained models for natural language processing: A survey (2020, background): “com/google-research/text-to-text-transfer-transformer ERNIE(Baidu) [164, 165] Framework: PaddlePaddle https://github.”
Evaluation of Text Generation: A Survey (2020, methodology): “…the performance of NLG using large-scale pre-trained language models for contextual word embeddings (Peters et al., 2018; Devlin et al., 2018; Sun et al., 2019; Dong et al., 2019), using better sampling methods to reduce degeneration in decoding (Zellers et al., 2019; Holtzman et al., 2020),…”
SemEval-2020 Task 12: Multilingual Offensive Language Identification in Social Media (OffensEval 2020) (2020, methodology): “For Subtasks B and C, they used knowledge distillation in a teacher-student framework, using Transformers such as ALBERT and ERNIE 2.0 (Sun et al., 2019) as teacher models, achieving an F1 score of 0.7462 and 0.7145, for Subtasks B and C respectively.”
RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering (2020, methodology): “0 base (Sun et al., 2019), and the cross-encoder is initialized with ERNIE 2.”
Finding Friends and Flipping Frenemies: Automatic Paraphrase Dataset Augmentation Using Graph Theory (2020, methodology): “According to the leaderboard of GLUE benchmark (Wang et al., 2018), an ALBERT based model (Lan et al., 2019) and ERNIE (Sun et al., 2019) are currently the top two models on QQP task with an accuracy of 91.0% and 90.9% on the original testing setComparing to these state-of-the-art models, we can rea”
A Survey on Knowledge Graphs: Representation, Acquisition, and Applications (2020, neutral): “0 [153] via continual multi-task learning.”
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜 (2021, methodology): “Initially developed as Chinese LMs, the ERNIE family [130, 131, 145] produced ERNIE-Gen, which was also trained on the original (English) BERT dataset, joining the ranks of very large LMs.”
PAIR: Leveraging Passage-Centric Similarity Relation for Improving Dense Passage Retrieval (2021, methodology): “DPR-E is our implementation of DPR using ERNIE (Sun et al., 2020)”
Exploring Text-transformers in AAAI 2021 Shared Task: COVID-19 Fake News Detection in English (2021, methodology): “[15] proposed continuous multi-task pre-training and several pre-training tasks in Ernie 2.0.”
NEZHA: Neural Contextualized Representation for Chinese Language Understanding (2019, methodology): “Pre-trained language models such as ELMo [2], BERT [1], ERNIE-Baidu [3, 4], ERNIE-Tsinghua [5], XLNet [6], RoBERTa [7] and MegatronLM 1 have demonstrated remarkable successes in modeling contextualized word representations by utilizing the massive amount of training text.”