KG-BERT: BERT for Knowledge Graph Completion

By Liang Yao, Chengsheng Mao, Yuan Luo. arXiv.org, 2019

821 citations recorded by the source corpus.

Knowledge graphs are important resources for many artificial intelligence tasks but often suffer from incompleteness. In this work, we propose to use pre-trained language models for knowledge graph completion. We treat triples in knowledge graphs as textual sequences and propose a novel framework named Knowledge Graph Bidirectional Encoder Representations from Transformer (KG-BERT) to model these triples. Our method takes entity and relation descriptions of a triple as input and computes scoring function of the triple with the KG-BERT language model. Experimental results on multiple benchmark knowledge graphs show that our method can achieve state-of-the-art performance in triple classification, link prediction and relation prediction tasks.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/31184789ef4c3084af930b1e0dede3215b4a9240

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

The language of proteins: NLP, machine learning & protein sequences (2021, background): “between entities in knowledge graphs, and indeed improves performance [78,115].”
A Review on Language Models as Knowledge Bases (2022, methodology): “Yao et al. (2019) also imbues entity and relation information by taking the entities and relations in a sequence as input into existing LMs. Aghajanyan et al. (2021, 2022) look at methods to incorporate more knowledge with data not commonly used in unimodal and multimodal settings.”
Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning (2020, methodology): “KG-BERT (Yao et al., 2019) directly concatenates the head, relation and tail of a triple, and finetunes pre-trained LMs on such data with binary classification objective, i.e., whether a triple is correct or not.”
Knowledge Graphs: Opportunities and Challenges (2023, methodology): “Furthermore, the knowledge could be extracted from unstructured documents (e.g., news, research papers, and patents) by adopting relation, entity, or attribute extraction methods (Liu et al. 2020; Yu et al. 2020; Yao et al. 2019).”
A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPT (2023, methodology): “A.5 Text Graph Year Method Task Code 2016 Li et al. [165] Text-to-KG Generation - 2019 KG-BERT [166] Text-to-KG Generation https://github.com/yao8839836/kg-bert 2020 Malaviya et al. [167] Text-to-KG Generation https://github.com/allenai/commonsense-kg-completion 2019 Petroni et al. citepetroni2019la”
Relational Memory-Augmented Language Models (2022, methodology): “This sequential representation can well capture the order of head entities and tail entities and is also adopted by KGBERT (Yao et al., 2019) and Kepler (Wang et al., 2021b).”
A Survey on Knowledge Graphs: Representation, Acquisition, and Applications (2020, methodology): “Similarly, KG-BERT [59] borrows the idea form language model pre-training and takes Bidirectional Encoder Representations from Transformer (BERT) model as an encoder for entities and relations.”
BERT-INT: A BERT-based Interaction Model For Knowledge Graph Alignment (2020, methodology): “For example, KG-BERT [Yao et al. , 2019] shows that on the task of triplet clas-siﬁcation, only using the descriptions of entities and relations can outperform the best KG embedding model by about 1-5% accuracy.”
A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunities (2022, background): “the KGBert [12] proposed by Alibaba surpasses humans in the field of FSL for the first time.”
Structure-Augmented Text Representation Learning for Efficient Knowledge Graph Completion (2020, methodology): “We empirically find such representation learning using the deterministic strategy is critical to the success of textual encoding KGC, consistent with previous works [26, 45].”
Deep Bidirectional Language-Knowledge Graph Pretraining (2022, neutral): “Several works additionally use textual data or pretrained LMs to help learn KG embeddings and link prediction (Riedel et al. 2013; Toutanova et al. 2015; Xie et al. 2016; Yao et al. 2019; Kim et al. 2020; Li et al. 2022).”
Relational World Knowledge Representation in Contextual Language Models: A Review (2021, background): “…2021; Guan et al., 2020) Lexicalizing paths (Clark et al., 2020; Talmor et al., 2020a,b) Linearized sequences (§ 5.2) Training on triple sequences (Yao et al., 2019; Agarwal et al., 2021) Injecting triples into text (Liu et al., 2020) Dedicated embeddings (§ 5.3) Pooling entity representations…”
Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, methodology): “[133] 2022 E LLMsasTextEncoders CoDEx[134] 2022 E LLMsasTextEncoders LMKE[135] 2022 E LLMsforJointTextandKGEmbedding kNN-KGE[136] 2022 E LLMsforJointTextandKGEmbedding LambdaKG[137] 2023 E + D + ED LLMsforJointTextandKGEmbedding LLM-augmentedKGcompletion KG-BERT[26] 2019 E JointEncoding MTL-KGC[138]”
ReMindRAG: Low-Cost LLM-Guided Knowledge Graph Traversal for Efficient RAG (2025, unknown): “Inspired by prior works [3, 50, 11, 13], in the automated knowledge graph construction framework, our designed heterogeneous knowledge graph consists of an entity node set E , an anchor node set A (storing text chunk titles), a text chunk set C , and relational connections.”