Deep Bidirectional Language-Knowledge Graph Pretraining

By Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D. Manning, Percy Liang, Jure Leskovec. 2022

279 citations recorded by the source corpus.

Pretraining a language model (LM) on text has been shown to help various downstream NLP tasks. Recent works show that a knowledge graph (KG) can complement text data, offering structured background knowledge that provides a useful scaffold for reasoning. However, these works are not pretrained to learn a deep fusion of the two modalities at scale, limiting the potential to acquire fully joint representations of text and KG. Here we propose DRAGON (Deep Bidirectional Language-Knowledge Graph Pretraining), a self-supervised approach to pretraining a deeply joint language-knowledge foundation model from text and KG at scale. Specifically, our model takes pairs of text segments and relevant KG subgraphs as input and bidirectionally fuses information from both modalities. We pretrain this model by unifying two self-supervised reasoning tasks, masked language modeling and KG link prediction. DRAGON outperforms existing LM and LM+KG models on diverse downstream tasks including question answering across general and biomedical domains, with +5% absolute gain on average. In particular, DRAGON achieves notable performance on complex reasoning about language and knowledge (+10% on questions involving long contexts or multi-step reasoning) and low-resource QA (+8% on OBQA and RiddleSense), and new state-of-the-art results on various BioNLP tasks. Our code and trained models are available at https://github.com/michiyasunaga/dragon.

Subfields: Artificial Intelligence, Molecular Biology, Large Language Models, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Large Language Models

In Knowledge Graphs & Representation

Citations in the corpus

Foundation models for generalist medical artificial intelligence (2023, background): “Approaches may build on previous work that combines language models and knowledge graphs 25,26 to reason step-by-step about surgical tasks.”
Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine (2023, methodology): “Models such as PubMedBERT [10], BioLinkBERT [37], DRAGON [36], BioGPT [19], and BioMedLM [2] were pretrained with self-supervised objectives us-ing domain-specific data sources, such as the PubMed corpus and UMLS knowledge graph.”
Retrieval-Augmented Multimodal Language Modeling (2023, background): “…a collection of text passages (Hashimoto et al., 2018; Khandelwal et al., 2019; Karpukhin et al., 2020; Guu et al., 2020; Lewis et al., 2020b;a; Yasunaga et al., 2022b; Borgeaud et al., 2022; Shi et al., 2023) or a structured knowledge base (Zhang et al., 2019; Agarwal et al., 2021; Xie et…”
Towards Expert-Level Medical Question Answering with Large Language Models (2023, unknown): “A majority of these approaches involve smaller language models trained using domain speciﬁc data (BioLinkBert [11], DRAGON [12], PubMedGPT [13], PubMedBERT [14], BioGPT [15]), resulting in a steady improvement in state-of-the-art performance on benchmark datasets such as MedQA (USMLE) [16], MedMCQA…”
Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Health (2023, methodology): “Non-LLM SOTA [64] Fine-tune 47.5 73.4 / - - Note: All numbers are accuracy in percentages.”
Large Language Models on Graphs: A Comprehensive Survey (2023, methodology): “DRAGON [81] further proposes strategies to pretrain GreaseLM with unsupervised signals.”
Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, methodology): “DRAGON [44] presents a self-supervised method to pre-train a joint language-knowledge foundation model from text and KG.”
Knowledge Graph Prompting for Multi-Document Question Answering (2023, background): “• Existing Knowledge Base: One common approach in the literature is to use existing knowledge bases or extract subgraphs from them for specific tasks (Yasunaga et al. 2022; Dong et al. 2023; Yasunaga et al. 2021).”
Retrieval And Structuring Augmented Generation with Large Language Models (2025, unknown): “Structure-enhanced LLM generation grounds model responses in explicit knowledge structures, reducing hallucinations and improving factual consistency.”
RAS: Retrieval-And-Structuring for Knowledge-Intensive LLM Generation (2025, unknown): “RAS addresses several critical limitations of existing approaches.”