ERNIE: Enhanced Language Representation with Informative Entities

By Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, Qun Liu. 2019

1,575 citations recorded by the source corpus.

Neural language representation models such as BERT pre-trained on large-scale corpora can well capture rich semantic patterns from plain text, and be fine-tuned to consistently improve the performance of various NLP tasks. However, the existing pre-trained language models rarely consider incorporating knowledge graphs (KGs), which can provide rich structured knowledge facts for better language understanding. We argue that informative entities in KGs can enhance language representation with external knowledge. In this paper, we utilize both large-scale textual corpora and KGs to train an enhanced language representation model (ERNIE), which can take full advantage of lexical, syntactic, and knowledge information simultaneously. The experimental results have demonstrated that ERNIE achieves significant improvements on various knowledge-driven tasks, and meanwhile is comparable with the state-of-the-art model BERT on other common NLP tasks. The code and datasets will be available in the future.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “…language representation learning (Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al., 2019; Song et al., 2019; Zhang et al., 2019; Yang et al., 2019), we propose several strategies that pre-train the encoder and de-coder before optimizing them jointly with the…”
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model (2022, methodology): “In particular, Radford et al. (2018); Devlin et al. (2019) demonstrated strong results using pretrained Transformer language models, prompting work on progressively better models (Liu et al., 2019; Yang et al., 2019; Lewis et al., 2020; Raffel et al., 2020; Zhang et al., 2019, etc.).”
How Can We Know What Language Models Know? (2019, methodology): “Similar extensions have been applied to pre-trained LMs like BERT, where contextualized representations are enhancedwith entity embeddings (Zhang et al., 2019; Peters et al., 2019; Pörner et al., 2019).”
Knowledge Enhanced Contextual Word Representations (2019, methodology): “As shown in Table 7, KnowBert-W+W shows an improvement of 0.6% F 1 over ERNIE and 2.5% over BERT BASE .”
Editing Factual Knowledge in Language Models (2021, background): “Approaches that incentivize memorization of factual knowledge show to be ben-eﬁcial for many downstream tasks suggesting that research on methods that effectively edit the memory of a model is indeed important (Zhang et al., 2019; Sun et al., 2019, 2020).”
Modifying Memories in Transformer Models (2020, background): “Another line of work (Sun et al., 2019; Zhang et al., 2019; Peters et al., 2019) focuses on leveraging the readily available structured KBs to further complement the knowledge possessed by language models.”
A Primer in BERTology: What We Know About How BERT Works (2020, methodology): “Liu et al. (2019c) explore the benefits 4New token-level tasks in ERNIE include prediction whether a token is capitalized and whether it occurs in other segments of the same document.”
Pre-Trained Models: Past, Present and Future (2021, background): “An interesting attempt is OAG-BERT (Liu et al., 2021a), which integrates heterogeneous structural knowledge in the open academic graph (OAG) (Zhang et al., 2019a), which covers 0.7 billion heterogeneous entities and 2 billion relations.”
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (2020, methodology): “For this task, we evaluate our models on Open Entity (Choi et al., 2018) and FIGER (Ling et al., 2015) following the same split setting as Zhang et al. (2019).”
MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension (2019, methodology): “The top system, which took advantage of newer pre-trained language models (Yang et al., 2019; Zhang et al., 2019), achieved an average F1 score of 72.”
Self-Supervised Learning: Generative or Contrastive (2020, background): “ERNIE [114], [149] NLP G AE Masked words Sentence”
Pre-trained models for natural language processing: A survey (2020, background): “The external knowledge ranges from linguistic [94, 83, 136, 191], semantic [99], commonsense [52], factual [214, 136, 111, 202, 195], to domain-specific knowledge [58, 111].”
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation (2019, background): “These embeddings not only help with the KG completion task, but can also improve the performance of various NLP applications (Zaremoodi et al., 2018; Zhong et al., 2019).”
K-BERT: Enabling Language Representation with Knowledge Graph (2019, methodology): “While the pre-trained LR model is an emerging direction, there is little work on its fusion with KG. THU-ERNIE (Zhang et al. 2019) is a pioneer in this direction by fusing entity information, but the relations between entities are ignored by it.”