Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning

By Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, Weizhu Chen. 2020

72 citations recorded by the source corpus.

In this work, we aim at equipping pre-trained language models with structured knowledge. We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme that exploits relational knowledge underlying the text. This is fulfilled by using a linked knowledge graph to select informative entities and then masking their mentions. In addition we use knowledge graphs to obtain distractors for the masked entities, and propose a novel distractor-suppressed ranking objective which is optimized jointly with masked language model. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training, to inject language models with structured knowledge via learning from raw text. It is more efficient than retrieval-based methods that perform entity linking and integration during finetuning and inference, and generalizes more effectively than the methods that directly learn from concatenated graph triples. Experiments show that our proposed model achieves improved performance on five benchmark datasets, including question answering and knowledge base completion tasks.

Subfields: Artificial Intelligence, Natural Language Processing, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Graph Learning

Citations in the corpus

DeBERTa: Decoding-enhanced BERT with Disentangled Attention (2020, methodology): “These PLMs have been fine-tuned using task-specific labels and created new state-of-the-art in many downstream natural language processing (NLP) tasks [13, 14, 15, 16, 17, 18].”
Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “ChatGPT/GPT-4 are LLM-based chatbots that can communicate with humans in a natural dialogue format.”
Relational World Knowledge Representation in Contextual Language Models: A Review (2021, background): “Shen et al. (2020) train LMs to internalize world knowledge by measuring the pairwise com-”
Knowledge-Augmented Methods for Natural Language Processing (2023, methodology): “…can be categorized into explicit methods via concept/entity embeddings (Zhang et al., 2019; Peters et al., 2019; Liu et al., 2020; Yu et al., 2020a; Zeng et al., 2020) and implicit methods via entity masking prediction (Sun et al., 2019; Shen et al., 2020; Xiong et al., 2020; Wang et al., 2019).”
GreaseLM: Graph REASoning Enhanced Language Models for Question Answering (2022, background): “However, much like for QA, the modality interaction is typically limited to knowledge feeding language (Zhang et al., 2019; Shen et al., 2020; Yu et al., 2020), rather than designing interactions across multiple layers.”
Deep Bidirectional Language-Knowledge Graph Pretraining (2022, unknown): “…et al. 2019; Rosset et al. 2020); Some works use the KG entity information or structure to create additional training signals (Xiong et al. 2020; Shen et al. 2020; Wang et al. 2021; Liu et al. 2021; Yu et al. 2022; Ke et al. 2021); Several works add KG triplet information directly to the LM…”
A Survey on Knowledge Graphs: Representation, Acquisition, and Applications (2020, background): “GLM [192] proposes a graph-guided entity masking scheme to utilize knowledge graph implicitly.”