LinkBERT: Pretraining Language Models with Document Links

By Michihiro Yasunaga, Jure Leskovec, Percy Liang. 2022

493 citations recorded by the source corpus.

Language model (LM) pretraining captures various knowledge from text corpora, helping downstream tasks. However, existing methods such as BERT model a single document, and do not capture dependencies or knowledge that span across documents. In this work, we propose LinkBERT, an LM pretraining method that leverages links between documents, e.g., hyperlinks. Given a text corpus, we view it as a graph of documents and create LM inputs by placing linked documents in the same context. We then pretrain the LM with two joint self-supervised objectives: masked language modeling and our new proposal, document relation prediction. We show that LinkBERT outperforms BERT on various downstream tasks across two domains: the general domain (pretrained on Wikipedia with hyperlinks) and biomedical domain (pretrained on PubMed with citation links). LinkBERT is especially effective for multi-hop reasoning and few-shot QA (+5% absolute improvement on HotpotQA and TriviaQA), and our biomedical LinkBERT sets new states of the art on various BioNLP tasks (+7% on BioASQ and USMLE). We release our pretrained models, LinkBERT and BioLinkBERT, as well as code and data.

Subfields: Artificial Intelligence, Molecular Biology, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

In Natural Language Processing

Citations in the corpus

Galactica: A Large Language Model for Science (2022, background): “On PubMedQA, we achieve a score of 77.6% which outperforms the state-of-the-art of 72.2% (Yasunaga et al., 2022).”
Large Language Models as Analogical Reasoners (2023, methodology): “This can include using reasoning-intensive data during training (Wu et al., 2021; Yasunaga et al., 2022b; Lightman et al., 2023; Moor et al., 2023), retrieving structured knowledge (Lin et al., 2019; Feng et al., 2020; Zhang et al., 2022a; Yasunaga et al., 2021; 2022a; Xie et al., 2022), and…”
Retrieval-Augmented Multimodal Language Modeling (2023, background): “…a collection of text passages (Hashimoto et al., 2018; Khandelwal et al., 2019; Karpukhin et al., 2020; Guu et al., 2020; Lewis et al., 2020b;a; Yasunaga et al., 2022b; Borgeaud et al., 2022; Shi et al., 2023) or a structured knowledge base (Zhang et al., 2019; Agarwal et al., 2021; Xie et…”
A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethics (2023, background): “[167].”
Deep Bidirectional Language-Knowledge Graph Pretraining (2022, unknown): “Similarly, as seen in §3.4.1, D RAGON also attains large gains over GreaseLM on complex questions containing negation, conjunction and prepositional phrases (Table 2), and extrapolates to questions more complex than seen in training sets (Figure 2).”
CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection (2023, methodology): “We can see that the CLIP-based embedding can significantly improve the performance comparing with conventional one-hot labels (DoDNet [88]) and text-only pre-trained embedding (BioLinkBERT [83]).”
Exploring the Potential of Large Language Models (LLMs)in Learning on Graphs (2023, methodology): “We empirically find that when we adopt deep sentence embedding models as enhancers at the feature level, they present good performance under different dataset split settings, and also scalability.”
Can Generalist Foundation Models Outcompete Special-Purpose Tuning? Case Study in Medicine (2023, methodology): “Models such as PubMedBERT [10], BioLinkBERT [37], DRAGON [36], BioGPT [19], and BioMedLM [2] were pretrained with self-supervised objectives us-ing domain-specific data sources, such as the PubMed corpus and UMLS knowledge graph.”
Towards Expert-Level Medical Question Answering with Large Language Models (2023, unknown): “A majority of these approaches involve smaller language models trained using domain speciﬁc data (BioLinkBert [11], DRAGON [12], PubMedGPT [13], PubMedBERT [14], BioGPT [15]), resulting in a steady improvement in state-of-the-art performance on benchmark datasets such as MedQA (USMLE) [16], MedMCQA…”
Can large language models reason about medical questions? (2022, unknown): “Fine-tuned BERT, BioLinkBERT 58 ; DPR, dense passage retrieval.”
BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining (2022, methodology): “…PubMedQA [16], BioASQ [30, 31] Document Classiﬁcation BERT [2], BlueBERT [8], SciBERT [18], SPECTER [32], PubMedBERT [9], BioELECTRA [28], LinkBERT [29] HoC [17], SciDocs [32] Document Classiﬁcation Document classiﬁcation is to classify a document into predeﬁned categories (single label or multi…”
Large Language Models on Graphs: A Comprehensive Survey (2023, methodology): “When extended in the graph domain, language model pretraining strategies include document relation prediction [30], network-contextualized masked language modeling [31], contrastive social prediction [32] and context graph prediction [33].”
Contrastive Learning Using Graph Embeddings for Domain Adaptation of Language Models in the Process Industry (2025, unknown): “Other studies focus on utilizing the graph structure of documents for learning document-level semantics, such as SPECTER (Cohan et al., 2020), LinkBERT (Yasunaga et al., 2022), and SciNCL (Ostendorff et al., 2022), which use citation information and document relations to fine-tune models.”