Language Models are Open Knowledge Graphs

By Chenguang Wang, Xiao Liu, Dawn Song. 2020

149 citations recorded by the source corpus.

This paper shows how to construct knowledge graphs (KGs) from pre-trained language models (e.g., BERT, GPT-2/3), without human supervision. Popular KGs (e.g, Wikidata, NELL) are built in either a supervised or semi-supervised manner, requiring humans to create knowledge. Recent deep language models automatically acquire knowledge from large-scale corpora via pre-training. The stored knowledge has enabled the language models to improve downstream NLP tasks, e.g., answering questions, and writing code and articles. In this paper, we propose an unsupervised method to cast the knowledge contained within language models into KGs. We show that KGs are constructed with a single forward pass of the pre-trained language models (without fine-tuning) over the corpora. We demonstrate the quality of the constructed KGs by comparing to two KGs (Wikidata, TAC KBP) created by humans. Our KGs also provide open factual knowledge that is new in the existing KGs. Our code and KGs will be made publicly available.

Subfields: Artificial Intelligence, Large Language Models, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Zhiyuan Liu
3. Mohit Bansal
4. Maosong Sun
5. Zhe Gan
6. Graham Neubig
7. Minlie Huang
8. Xipeng Qiu

In Large Language Models

1. Jie Tang
2. Minlie Huang
3. Zhiyuan Liu
4. Xiao Liu
5. Ming Ding
6. Zhe Gan
7. Xipeng Qiu
8. Lijuan Wang

In Knowledge Graphs & Representation

1. Phil Blunsom
2. Minlie Huang
3. Maosong Sun
4. Zhilin Yang
5. Jie Tang
6. Yang Liu
7. Xu Han
8. Zhiyuan Liu

Citations in the corpus

• GPT Understands, Too (2021, background): “In (Wang et al., 2020), the authors investigate the attention matrices to find that the attentions would also indicate knowledge triples contained in the context and thus develop an open knowledge graph construction framework.”
• An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA (2021, background): “Specifically, large-scale language models such as GPT-3 (Brown et al. 2020) have shown powerful abilities in NLP tasks, such as knowledge retrieval (Wang, Liu, and Song 2020) and question answering (Brown et al. 2020).”
• Pre-Trained Models: Past, Present and Future (2021, methodology): “(Wang et al., 2020a) investigates the knowledge stored in”
• Towards Zero-Label Language Learning (2021, background): “These models are known to be good at memorizing facts from training data and capable of performing as open knowledge bases (Petroni et al., 2019; Wang et al., 2020; Roberts et al., 2020; Carlini et al., 2021).”
• DeepStruct: Pretraining of Language Models for Structure Prediction (2022, methodology): “We additionally compare to MAMA using 3 https://github.com/dair-iitd/OpenIE-standalone BERTLARGE from (Wang et al., 2020), which proposes to leverage knowledge stored in attention matrices for OIE.”
• Are Large Pre-Trained Language Models Leaking Your Personal Information? (2022, background): “Previous works have shown that large PLMs contain a significant amount of knowledge, which can be recovered by querying PLMs with appropriate prompts (Petroni et al., 2019; Bouraoui et al., 2020; Jiang et al., 2020a,b; Wang et al., 2020).”
• Inference-Time Intervention: Eliciting Truthful Answers from a Language Model (2023, background): “Wang et al. (2021) construct high-quality knowledge graphs from LLMs without human supervision.”
• When large language models meet personalization: perspectives of challenges and opportunities (2023, unknown): “LLMs can also achieve the end-to-end construction [62], [50], [42], [63], [55] to directly build KGs from raw text.”
• A Review on Language Models as Knowledge Bases (2022, background): “Several works have already approached improving LMs through the lens of KBs: Petroni et al. (2019); Dhingra et al. (2021); Wang et al. (2020); Heinzerling and Inui (2021); Sung et al. (2021).”
• Relational Memory-Augmented Language Models (2022, background): “, 2020) do not have explicit representations for such information and rely on it being implicitly encoded in their parameters (Liu et al., 2019; Petroni et al., 2019; Wang et al., 2020).”
• Towards a Universal Continuous Knowledge Base (2020, background): “Recently, a number of works have been studied on what does the pre-trained language model learns (Petroni et al., 2019; Bouraoui et al., 2020; Rogers et al., 2020; Wang et al., 2020).”