CoLAKE: Contextualized Language and Knowledge Embedding

By Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, Zheng Zhang. 2020

223 citations recorded by the source corpus.

With the emerging branch of incorporating factual knowledge into pre-trained language models such as BERT, most existing models consider shallow, static, and separately pre-trained entity embeddings, which limits the performance gains of these models. Few works explore the potential of deep contextualized knowledge representation when injecting knowledge. In this paper, we propose the Contextualized Language and Knowledge Embedding (CoLAKE), which jointly learns contextualized representation for both language and knowledge with the extended MLM objective. Instead of injecting only entity embeddings, CoLAKE extracts the knowledge context of an entity from large-scale knowledge bases. To handle the heterogeneity of knowledge context and language context, we integrate them in a unified data structure, word-knowledge graph (WK graph). CoLAKE is pre-trained on large-scale WK graphs with the modified Transformer encoder. We conduct experiments on knowledge-driven tasks, knowledge probing tasks, and language understanding tasks. Experimental results show that CoLAKE outperforms previous counterparts on most of the tasks. Besides, CoLAKE achieves surprisingly high performance on our synthetic task called word-knowledge graph completion, which shows the superiority of simultaneously contextualizing language and knowledge representation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Maosong Sun
3. Percy Liang
4. Minlie Huang
5. Xipeng Qiu
6. Erik Cambria
7. Philip S. Yu
8. Mohit Bansal

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Jie Zhou
3. Zhiyuan Liu
4. Maosong Sun
5. Percy Liang
6. Philip S. Yu
7. Luc Van Gool
8. Yang Liu

Citations in the corpus

• Pre-Trained Models: Past, Present and Future (2021, background): “Many works try to enhance PTMs by integrating entity and relation embeddings (Zhang et al., 2019b; Liu et al., 2020a; Peters et al., 2019; Sun et al., 2020; Rosset et al., 2020; Qin et al., 2021) or their alignments with the text (Xiong et al.”
• Pre-trained models for natural language processing: A survey (2020, background): “Text MLM+ERD WikiEn + Wikidata =BERT Yes CoLAKE [81] Transformer Enc.”
• A Survey on Knowledge Graphs: Representation, Acquisition, and Applications (2020, background): “CoLAKE [193] further exploits the knowledge context of an entity through a unified word-knowledge graph and a modified Transformer encoder.”
• ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning (2020, background): “Some enforce PLMs to memorize information about real-world entities and propose novel pretraining objectives (Xiong et al., 2019; Wang et al., 2019; Sun et al., 2020; Yamada et al., 2020).”
• Unifying Large Language Models and Knowledge Graphs: A Roadmap (2023, background): “Specifically, entities that can be reached within a certain number of hops are [107] 2020 E IntegratingKGsintoLanguageModelInputs ERNIE3.”
• OAG-BERT: Pre-train Heterogeneous Entity-augmented Academic Language Models (2021, background): “While many works have been concentrating on injecting the homogeneous entity and relation knowledge from large scale knowledge graphs [36, 40, 51], none of them considers the problem of heterogeneous scenario which usually holds true in practice.”
• Deep Bidirectional Language-Knowledge Graph Pretraining (2022, unknown): “One line of research aims to add entity features to LMs [12, 23, 24]; Some works use the KG entity information or structure to create additional training signals [13, 25, 14, 26, 27, 28]; Several works add KG triplet information directly to the LM input [29, 16, 15, 30, 31].”
• GreaseLM: Graph REASoning Enhanced Language Models for Question Answering (2022, background): “Sun et al. (2020)’s work is perhaps most similar, but they do not use the same interaction bottleneck, requiring high-precision entity mention spans for linking, and they limit expressivity through shared modality parameters for the LM and KG.”