Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training

By Oshin Agarwal, Heming Ge, Siamak Shakeri, Rami Al-Rfou. 2021

208 citations recorded by the source corpus.

Prior work on Data-To-Text Generation, the task of converting knowledge graph (KG) triples into natural text, focused on domain-specific benchmark datasets. In this paper, however, we verbalize the entire English Wikidata KG, and discuss the unique challenges associated with a broad, open-domain, large-scale verbalization. We further show that verbalizing a comprehensive, encyclopedic KG like Wikidata can be used to integrate structured KGs and natural language corpora. In contrast to the many architectures that have been developed to integrate these two sources, our approach converts the KG into natural text, allowing it to be seamlessly integrated into existing language models. It carries the further advantages of improved factual accuracy and reduced toxicity in the resulting language model. We evaluate this approach by augmenting the retrieval corpus in a retrieval language model and showing significant improvements on the knowledge intensive tasks of open domain QA and the LAMA knowledge probe.

Subfields: Artificial Intelligence, Large Language Models, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Knowledge Graphs & Representation

Citations in the corpus

UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Models (2022, methodology): “Structured knowledge encoding methods have been explored in many prior studies (Bogin et al., 2019; Lin et al., 2019; Yin et al., 2020a; Herzig et al., 2020; Agarwal et al., 2020; Saxena et al., 2020; Yasunaga and Liang, 2020; Yasunaga et al., 2021; Oguz et al., 2021).”
Dataset Debt in Biomedical Language Modeling (2022, unknown): “%) or were dead links (5%).”
ReGen: Reinforcement Learning for Text and Knowledge Base Generation using Pretrained Language Models (2021, methodology): “We will also present results for the TEKGEN Corpus (Agarwal et al., 2021) to show performance on a different, much larger dataset.”
Retrieval-Augmented Generation for AI-Generated Content: A Survey (2024, unknown): “Toutanova et al. [222] substituted the text corpus in REALM with subgraphs from a knowledge graph, yielding impressive results.”
KB-VLP: Knowledge Based Vision and Language Pretraining (2021, background): “…using knowledge during pretraining of language models (Yu et al., 2020; Xu et al., 2021; Ros-set et al., 2021; Zhou et al., 2020; He et al., 2020a; Xiong et al., 2019; He et al., 2020b; Agarwal et al., 2021), most have not been applied to multi-modal transformers (e.g. for vision and language).”
UniK-QA: Unified Representations of Structured and Unstructured Knowledge for Open-Domain Question Answering (2020, methodology): “Concurrent work (Agarwal et al., 2021) proposed a similar idea for language model pre-training and also evaluated on open-domain QA.”
Retrieval-Augmented Multimodal Language Modeling (2023, background): “, 2022) or a structured knowledge base (Zhang et al., 2019; Agarwal et al., 2021; Xie et al., 2022; Yasunaga et al., 2021; 2022a).”
Iteratively Prompt Pre-trained Language Models for Chain of Thought (2022, methodology): “For 2Wiki, we ﬁlter from the processed Wikidata triples provided by (Agarwal et al., 2021) vided knowledge statements are in natural language and it’s hard to retrieve high quality knowledge statements as such.”
LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks (2022, background): “Several works also extend the existing LMs to handle different input data types, such as images [75, 76], audio [77], tabular data [78], and knowledge base [79] by updating the pretraining phase with these data and their corresponding tasks or using general-purpose architecture [80].”
Deep Bidirectional Language-Knowledge Graph Pretraining (2022, unknown): “How to effectively combine text and KGs for pretraining is an open problem and presents challenges.”
DeepStruct: Pretraining of Language Models for Structure Prediction (2022, methodology): “In particular, we use T-REx (ElSahar et al., 2018), TEKGEN and KELM (Agarwal et al., 2021), WebNLG (Gardent et al., 2017), Concept-Net (Speer and Havasi, 2012).”