ERNIE: Enhanced Representation through Knowledge Integration

By Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Xuyi Chen, Han Zhang, Xin Tian, Danxiang Zhu, Hao Tian, Hua Wu. 2019

1,025 citations recorded by the source corpus.

We present a novel language representation model enhanced by knowledge called ERNIE (Enhanced Representation through kNowledge IntEgration). Inspired by the masking strategy of BERT, ERNIE is designed to learn language representation enhanced by knowledge masking strategies, which includes entity-level masking and phrase-level masking. Entity-level strategy masks entities which are usually composed of multiple words.Phrase-level strategy masks the whole phrase which is composed of several words standing together as a conceptual unit.Experimental results show that ERNIE outperforms other baseline methods, achieving new state-of-the-art results on five Chinese natural language processing tasks including natural language inference, semantic similarity, named entity recognition, sentiment analysis and question answering. We also demonstrate that ERNIE has more powerful knowledge inference capacity on a cloze test.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

RoBERTa: A Robustly Optimized BERT Pretraining Approach (2019, methodology): “However, newer methods have improved performance by multi-task ﬁne tuning (Dong et al., 2019), incorporating entity embeddings (Sun et al., 2019), span prediction (Joshi et al., 2019), and multiple variants of autoregressive pretraining (Song et al., 2019; Chan et al., 2019; Yang et al., 2019).”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, background): “While previous and concurrent work has looked at masking (Sun et al., 2019) or dropping (Song et al., 2019; Chan et al., 2019) multiple words from the input – particularly as pretraining for language generation tasks – SpanBERT pretrains span representations (Lee et al., 2016), which are widely…”
KERMIT: Generative Insertion-Based Modeling for Sequences (2019, methodology): “Then, like its friends ELMo (Peters et al., 2018), BERT (Devlin et al., 2019), and ERNIE (Sun et al., 2019), we can also use KERMIT for self-supervised representation learning for use in downstream NLP tasks.”
DeBERTa: Decoding-enhanced BERT with Disentangled Attention (2020, background): “Since 2018, we have seen the rise of a set of large-scale Transformer-based Pre-trained Language Models (PLMs), such as GPT [2, 3], BERT [4], RoBERTa [5], XLNet [6], UniLM [7], ELECTRA [8], T5 [9], ALUM [10], StructBERT [11] and ERINE [12] .”
ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graph (2020, methodology): “Motivated by pre-trained models like BERT [1] and ERNIE [2] which have signiﬁcantly improved the performance on many NLP tasks, researchers ( [3] [4] [5] [6] [7] [8] [9] [10]) have noticed the importance of pre-training for vision-language tasks, e.g., Visual Question Answering(VQA) [11] and Visual…”
CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation (2021, methodology): “enization, which aims to avoid masking partial subtokens and is shown to be helpful (Sun et al., 2019).”
Editing Factual Knowledge in Language Models (2021, background): “Approaches that incentivize memorization of factual knowledge show to be ben-eﬁcial for many downstream tasks suggesting that research on methods that effectively edit the memory of a model is indeed important (Zhang et al., 2019; Sun et al., 2019, 2020).”
Modifying Memories in Transformer Models (2020, background): “Another line of work (Sun et al., 2019; Zhang et al., 2019; Peters et al., 2019) focuses on leveraging the readily available structured KBs to further complement the knowledge possessed by language models.”
CLUE: A Chinese Language Understanding Evaluation Benchmark (2020, background): “• ERNIE-base (Sun et al., 2019c) extends BERT-base with additional training data and leverages knowledge from Knowledge Graphs.”
ERNIE: Enhanced Language Representation with Informative Entities (2019, methodology): “It is a coincidence that both Sun et al. (2019) and we chose ERNIE as the model names, which follows the interesting naming habits like ELMo and BERT.”
A Primer in BERTology: What We Know About How BERT Works (2020, background): “Masking phrases and named entities (Sun et al., 2019b) improves representation of structured knowledge.”
Pre-Trained Models: Past, Present and Future (2021, methodology): “ERNIE-M (Ouyang et al., 2020) proposes back-translation masked language modeling (BTMLM), and expands the scale of parallel corpora through back-translation mechanisms.”
ERNIE 2.0: A Continual Pre-training Framework for Language Understanding (2019, background): “Pre-trained language representations such as ELMo(Peters et al. 2018), OpenAI GPT(Radford et al. 2018), BERT (Devlin et al. 2018), ERNIE 1.0 (Sun et al. 2019) 1 and XLNet(Yang et al. 2019) have been proven to be effective for improving the performances of various natural language understanding…”
OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework (2022, background): “A number of studies have been progressively advancing pretraining by improving pretraining tasks and designing more sophisticated model architectures [25, 26, 27, 28, 29, 30, 31].”
FastBERT: a Self-distilling BERT with Adaptive Inference Time (2020, methodology): “PKD-BERT (Sun et al., 2019a) adopts an incremental extraction process that learns generalizations from intermediate layers of the teacher model.”