Knowledge-Aware Language Model Pretraining

By Corby Rosset, Chenyan Xiong, Minh Hieu Phan, Xia Song, Paul N. Bennett, Saurabh Tiwary. 2020

93 citations recorded by the source corpus.

How much knowledge do pretrained language models hold? Recent research observed that pretrained transformers are adept at modeling semantics but it is unclear to what degree they grasp human knowledge, or how to ensure they do so. In this paper we incorporate knowledge-awareness in language model pretraining without changing the transformer architecture, inserting explicit knowledge layers, or adding external storage of semantic information. Rather, we simply signal the existence of entities to the input of the transformer in pretraining, with an entity-extended tokenizer; and at the output, with an additional entity prediction task. Our experiments show that solely by adding these entity signals in pretraining, significantly more knowledge is packed into the transformer parameters: we observe improved language modeling accuracy, factual correctness in LAMA knowledge probing tasks, and semantics in the hidden representations through edge probing.We also show that our knowledge-aware language model (KALM) can serve as a drop-in replacement for GPT-2 models, significantly improving downstream tasks like zero-shot question-answering with no task-related training.

Subfields: Artificial Intelligence, Large Language Models, Knowledge Graphs & Representation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Yejin Choi
3. Xiang Ren
4. Mohit Bansal
5. Minlie Huang
6. Jianfeng Gao
7. Maosong Sun
8. Yang Liu

In Large Language Models

1. Zhiyuan Liu
2. Yejin Choi
3. Minlie Huang
4. Jie Tang
5. Xiang Ren
6. Jiawei Han
7. Xu Han
8. Bill Yuchen Lin

In Knowledge Graphs & Representation

1. Xiang Ren
2. Yejin Choi
3. Ronan Le Bras
4. Shirui Pan
5. Bill Yuchen Lin
6. Maarten Sap
7. Daniel Fried
8. Xindong Wu

Citations in the corpus

• Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “…architectures on text with entity coreference information yields more entity-aware LLMs, both in recurrent (Henaff et al., 2017; Ji et al., 2017; Yang et al., 2017; Liu et al., 2019) and Transformer-based models (Févry et al., 2020; De Cao et al., 2020; Rosset et al., 2020; Zhang et al., 2022c).”
• Pre-Trained Models: Past, Present and Future (2021, background): “Many works try to enhance PTMs by integrating entity and relation embeddings (Zhang et al., 2019b; Liu et al., 2020a; Peters et al., 2019; Sun et al., 2020; Rosset et al., 2020; Qin et al., 2021) or their alignments with the text (Xiong et al.”
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining (2021, background): “This significantly boostes the generalization ability of the pretrained models in semantic-centric tasks, including more accurate question answering and more factually correct language generation (Guu et al., 2020; Roberts et al., 2020; Rosset et al., 2020).”
• Deep Bidirectional Language-Knowledge Graph Pretraining (2022, unknown): “…et al. 2019; Peters et al. 2019; Rosset et al. 2020); Some works use the KG entity information or structure to create additional training signals (Xiong et al. 2020; Shen et al. 2020; Wang et al. 2021; Liu et al. 2021; Yu et al. 2022; Ke et al. 2021); Several works add KG triplet information…”
• Relational World Knowledge Representation in Contextual Language Models: A Review (2021, background): “Other approaches leave the base LM’s internal architecture completely unchanged and focus only on aligning the LM’s word and entity embedding spaces at the input level (Rosset et al., 2020; Poerner et al., 2020).”
• Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “Some researchers have proposed to incorporate KGs into LLMs during the pre-training stage, which can help LLMs learn knowledge from KGs [35], [91].”
• A Review on Language Models as Knowledge Bases (2022, methodology): “Other models such as ERNIE , KEPLER , and KALM use existing algorithms to embed entity and relation knowledge into embeddings (Zhang et al., 2019; Wang et al., 2021b; Rosset et al., 2020).”
• Pre-training Text-to-Text Transformers for Concept-centric Common Sense (2020, methodology): “Some major differences include that (1) Our method employs sentence-level distractors that are in general grammatically correct but not in line with commonsense, thus require the model to perform relational commonsense reasoning while RTD is a token-level discrimination task and can often be solved ”