K-BERT: Enabling Language Representation with Knowledge Graph

By Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, Ping Wang. 2019

930 citations recorded by the source corpus.

Pre-trained language representation models, such as BERT, capture a general language representation from large-scale corpora, but lack domain-specific knowledge. When reading a domain text, experts make inferences with relevant knowledge. For machines to achieve this capability, we propose a knowledge-enabled language representation model (K-BERT) with knowledge graphs (KGs), in which triples are injected into the sentences as domain knowledge. However, too much knowledge incorporation may divert the sentence from its correct meaning, which is called knowledge noise (KN) issue. To overcome KN, K-BERT introduces soft-position and visible matrix to limit the impact of knowledge. K-BERT can easily inject domain knowledge into the models by being equipped with a KG without pre-training by itself because it is capable of loading model parameters from the pre-trained BERT. Our investigation reveals promising results in twelve NLP tasks. Especially in domain-specific tasks (including finance, law, and medicine), K-BERT significantly outperforms BERT, which demonstrates that K-BERT is an excellent choice for solving the knowledge-driven problems that require experts.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, methodology): “To encode structured knowledge, prior work linearized the structured knowledge and concatenated it with the text (Hwang et al., 2019; Liu et al., 2020; Hosseini-Asl et al., 2020; Liu et al., 2021), which has been augmented by positional encoding (e.g., row/column embedding) (Herzig et al., 2020;…”
FastBERT: a Self-distilling BERT with Adaptive Inference Time (2020, neutral): “K-BERT (Liu et al., 2020) injects knowledge graph into BERT model.”
A Survey on Knowledge Graphs: Representation, Acquisition, and Applications (2020, background): “ntities. ERNIE-Tsinghua [168] fuses informative entities via aggregated pre-training and random masking. BERT-MK [169] encodes graph contextualized knowledge and focuses on the medical corpus. K-BERT [170] infuses domain knowledge into BERT contextual encoder. ERNIEBaidu [171] introduces named entit”
Knowledge-Augmented Methods for Natural Language Processing (2023, methodology): “• Knowledge-augmented NLU: (Zhang et al., 2019; Peters et al., 2019; Liu et al., 2020; Ding et al., 2019; Lv et al., 2020; Yu et al., 2022b); • Knowledge-augmented NLG: (Zhou et al., 2018; Zhang et al., 2020a; Ji et al., 2020b; Lewis et al., 2020; Wang et al., 2021); • Commonsense Knowledge and…”
Generative artificial intelligence: a systematic review and applications (2024, neutral): “Liu et al. [81] introduce K-BERT, a novel approach that empowers language representation with knowledge graphs, enabling the incorporation of commonsense and domain-specific knowledge.”
CoLAKE: Contextualized Language and Knowledge Embedding (2020, methodology): “CoLAKE is conceptually similar to K-BERT (Liu et al., 2020) and BERT-MK (He et al., 2019).”
TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data (2020, methodology): “…contextual word representations, either by fusing vector representations of entities and relations on KBs into word representations of LMs (Peters et al., 2019; Zhang et al., 2019b,c), or by encouraging the LM to recover KB entities and relations from text (Sun et al., 2019; Liu et al., 2019a).”
OAG-BERT: Pre-train Heterogeneous Entity-augmented Academic Language Models (2021, background): “Previous approaches [28, 51] mainly focus on injecting homogeneous entities and relations from knowledge graphs likeWikidata, and very few of them look into situations where there are heterogeneous entities.”
Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “However, such direct knowledge triple concatenation method allows the tokens in the sentence to intensively interact with the tokens in the knowledge sub-graph, which could result in Knowledge Noise [36].”
Deep Bidirectional Language-Knowledge Graph Pretraining (2022, neutral): “…to create additional training signals (Xiong et al. 2020; Shen et al. 2020; Wang et al. 2021; Liu et al. 2021; Yu et al. 2022; Ke et al. 2021); Several works add KG triplet information directly to the LM input (Liu et al. 2020; Sun et al. 2021; Agarwal et al. 2021; Sun et al. 2020; He et al. 2020).”