KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation

By Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhiyuan Liu, Juan-Zi Li, Jian Tang. 2019

843 citations recorded by the source corpus.

Abstract Pre-trained language representation models (PLMs) cannot well capture factual knowledge from text. In contrast, knowledge embedding (KE) methods can effectively represent the relational facts in knowledge graphs (KGs) with informative entity embeddings, but conventional KE models cannot take full advantage of the abundant textual information. In this paper, we propose a unified model for Knowledge Embedding and Pre-trained LanguagERepresentation (KEPLER), which can not only better integrate factual knowledge into PLMs but also produce effective text-enhanced KE with the strong PLMs. In KEPLER, we encode textual entity descriptions with a PLM as their embeddings, and then jointly optimize the KE and language modeling objectives. Experimental results show that KEPLER achieves state-of-the-art performances on various NLP tasks, and also works remarkably well as an inductive KE model on KG link prediction. Furthermore, for pre-training and evaluating KEPLER, we construct Wikidata5M1 , a large-scale KG dataset with aligned entity descriptions, and benchmark state-of-the-art KE methods on it. It shall serve as a new KE benchmark and facilitate the research on large KG, inductive KE, and KG with text. The source code can be obtained from https://github.com/THU-KEG/KEPLER.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

ProoFVer: Natural Logic Theorem Proving for Fact Verification (2021, methodology): “We first fine tune KEPLER with the FEVER training dataset.”
Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs (2021, methodology): “We also construct (4) a sequence prediction task using data from Wikidata5m, which is a relational knowledge base with over 20 million triplets (Wang et al., 2021b).”
K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapters (2020, methodology): “Other baseline models, such as BERT-base (Zhang et al., 2019), ERNIE (Zhang et al., 2019), KnowBERT (Peters et al., 2019), KEPLER (Wang et al., 2019) and RoBERTa+multitask are described in Section 2 and 4.1.”
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (2019, methodology): “Two other knowledge-enhanced MLMs are KEPLER (Wang et al., 2019c) and K-Adapter (Wang et al., 2020), which are based on Roberta (Liu et al., 2019) rather than BERT.”
CPM: A Large-scale Generative Chinese Pre-trained Language Model (2020, background): “ERNIE (Zhang et al., 2019), KEPLER (Wang et al., 2019) and SentiLARE (Ke et al., 2020) introduce external knowledge to language representation learning by auxiliary pre-training tasks.”
CoLAKE: Contextualized Language and Knowledge Embedding (2020, methodology): “In contrast, KEPLER (Wang et al., 2019c) aims to beneﬁt both sides so jointly learn language model and knowledge embedding.”
Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (2023, background): “Require: Input x , LLM π , depth limit D max sample limit N .”
QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering (2021, methodology): “Several works study joint representations of external textual knowledge (e.g., Wikipedia articles) and structured knowledge (e.g., KGs) (Riedel et al., 2013; Toutanova et al., 2015; Xiong et al., 2019; Sun et al., 2019; Wang et al., 2019b).”
A Survey of Knowledge Graph Reasoning on Graph Types: Static, Dynamic, and Multi-Modal (2022, background): “WD-singer [210] and wikidata5m [216] are derived from it according to different scopes.”
LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention (2020, methodology): “XLNet+Veriﬁer (Li et al., 2019) (Zhang et al., 2018a) 44.1 45.4 45.4 46.7 BERT (Wang et al., 2019a) - - 71.3 72.0 XLNet+Veriﬁer (Li et al., 2019) 80.6 82.1 81.5 82.7 RoBERTa (Liu et al., 2020) 89.0 89.5 - - RoBERTa (ensemble) (Liu et al., 2020) based on this dataset (Ostermann et al., 2019).”
A Review on Language Models as Knowledge Bases (2022, methodology): “, 2021), while other datasets have been adapted for that purpose (Levy et al., 2017; Wang et al., 2021c).”
ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning (2020, background): “Some enforce PLMs to memorize information about real-world entities and propose novel pretraining objectives (Xiong et al., 2019; Wang et al., 2019; Sun et al., 2020; Yamada et al., 2020).”