Large Scale Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training

By Oshin Agarwal, Heming Ge, Siamak Shakeri, Rami Al-Rfou. 2020

36 citations recorded by the source corpus.

Generating natural sentences from Knowledge Graph (KG) triples, known as Data-To-Text Generation, is a task with many datasets for which numerous complex systems have been developed. However, no prior work has attempted to perform this generation at scale by converting an entire KG into natural text. In this paper, we verbalize the entire Wikidata KG, and create a KG-Text aligned corpus in the training process. We discuss the challenges in verbalizing an entire KG versus verbalizing smaller datasets. We further show that verbalizing an entire KG can be used to integrate structured and natural language data. In contrast to the many architectures that have been developed to integrate the structural differences between these two sources, our approach converts the KG into the same format as natural text allowing it to be seamlessly plugged into existing natural language systems. We evaluate this approach by augmenting the retrieval corpus in REALM and showing improvements, both on the LAMA knowledge probe and open domain QA.

Subfields: Artificial Intelligence, Natural Language Processing, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jie Tang
2. Yuxiao Dong
3. Xiao Liu
4. Ming Ding
5. Zhengxiao Du
6. Md. Rashad Al Hasan Rony
7. W. Tam
8. Wenguang Chen

In Natural Language Processing

1. Jie Tang
2. Ming Ding
3. Yuxiao Dong
4. W. Tam
5. Wenguang Chen
6. Aohan Zeng
7. Xiao Liu
8. Ricardo Usbeck

In Knowledge Graphs & Representation

1. Jie Tang
2. Danai Koutra
3. Tara Safavi

Citations in the corpus

• GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “For KELM data, since it is based on the full Wikidata schema (which contains too many relations to be enumerated), we create two KELM-specific prompts for the task of Relation Extraction and Knowledge Slot Filling: (Relation Extraction, Prompt 1, KELM ONLY) {# kelm”
• Relational World Knowledge Representation in Contextual Language Models: A Review (2021, background): “Training on triples only Agarwal et al. (2021) train T5 on subject-relation-object surface forms in order to output natural language sentences expressing these triples—i.”
• RoMe: A Robust Metric for Evaluating Natural Language Generation (2022, methodology): “…metric by conducting experiments on092 datasets from various domains of NLG such093 as knowledge graph based language generation094 dataset (KELM (Agarwal et al., 2021)), dialogue095 datasets (Eric et al., 2017; Chaudhuri et al., 2021),096 the NLG2017 challenge dataset (Shimorina et al.,097…”