Knowledge Graph and Text Jointly Embedding

By Zhen Wang, Jianwen Zhang, Jianlin Feng, Zheng Chen. 2014

401 citations recorded by the source corpus.

We examine the embedding approach to reason new relational facts from a largescale knowledge graph and a text corpus. We propose a novel method of jointly embedding entities and words into the same continuous vector space. The embedding process attempts to preserve the relations between entities in the knowledge graph and the concurrences of words in the text corpus. Entity names and Wikipedia anchors are utilized to align the embeddings of entities and words in the same space. Large scale experiments on Freebase and a Wikipedia/NY Times corpus show that jointly embedding brings promising improvement in the accuracy of predicting facts, compared to separately embedding knowledge graphs and text. Particularly, jointly embedding enables the prediction of facts containing entities out of the knowledge graph, which cannot be handled by previous embedding methods. At the same time, concerning the quality of the word embeddings, experiments on the analogical reasoning task show that jointly embedding is comparable to or slightly better than word2vec (Skip-Gram).

Subfields: Artificial Intelligence, Natural Language Processing, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Graph Learning

Citations in the corpus

Knowledge Enhanced Contextual Word Representations (2019, methodology): “These methods broadly fall into two categories: translational distance models (Bordes et al., 2013; Wang et al., 2014b; Lin et al., 2015; Xiao et al., 2016) which use a distance-based scoring function, and linear models (Nickel et al., 2011; Yang et al., 2014; Trouillon et al., 2016; Dettmers et…”
ERNIE: Enhanced Language Representation with Informative Entities (2019, background): “In fact, some work has attempted to joint representation learning of words and entities for effectively leveraging external KGs and achieved promising results (Wang et al., 2014; Toutanova et al., 2015; Han et al., 2016; Yamada et al., 2016; Cao et al., 2017, 2018).”
Pre-trained models for natural language processing: A survey (2020, background): “Early studies [197, 217, 201, 205] focused on learning knowledge graph embeddings and word embedding jointly.”
Representing Text for Joint Embedding of Text and Knowledge Bases (2015, methodology): “Wang et al. (2014a) combined knowledge base and text information by embedding knowledge base entities and the words in their names in the same vector space, but did not model the textual co-occurrences of entity pairs and the expressed textual relations.”
K-BERT: Enabling Language Representation with Knowledge Graph (2019, methodology): “Wang et al. (2014) proposed a novel method of jointly embedding entities and words into the same continuous vector space basing on the idea of word2vec (Mikolov et al. 2013).”
Joint Learning of the Embedding of Words and Entities for Named Entity Disambiguation (2016, result): “Particularly, Wang et al. (Wang et al., 2014) have recently revealed that the joint modeling of the embedding of words and entities can improve performance in several tasks including the link prediction task, which is somewhat analogous to our experimental results.”
Incorporating Knowledge Graph Embeddings into Topic Modeling (2017, neutral): “Knowledge graph embedding aims at embedding entities and relationships of knowledge graphs in vector spaces (Bordes et al. 2011; 2013; Wang et al. 2014b; 2014a; Guo et al. 2015; Xie et al. 2016).”
KG-BERT: BERT for Knowledge Graph Completion (2019, methodology): “Text-enhanced models DKRL and SSP can also outperform structure only methods TransE and TransH.”
Distant Supervision for Relation Extraction with Sentence-Level Attention and Entity Descriptions (2017, background): “Another line is to introduce external semantic repositories such as knowledge graphs (Weston et al. 2013; Wang et al. 2014).”
A Survey on Knowledge Graphs: Representation, Acquisition, and Applications (2020, methodology): “[56] used a three-component loss L = LK +LT +LA of knowledge model LK , text model LT and alignment model LA.”
Knowledge Graph Embedding Based Question Answering (2019, methodology): “Attempts [37, 38, 50] have also been made to apply TransE and word2vec [28] to model a KG and a corpus respectively, and then fuse them based on anchors in Wikipedia [38], entity descriptions [50], or contextual words of predicates/entities learned from the corpus [37].”
Multi-Task Feature Learning for Knowledge Graph Enhanced Recommendation (2019, methodology): “ified into the following two categories: (1) Translational distance models exploit distance-based scoring functions when learning representations of entities and relations, such as TransE [2], TransH [35], and TransR [13]; (2) Semantic matching models measure plausibility of knowledge triples by mat”
Structure-Augmented Text Representation Learning for Efficient Knowledge Graph Completion (2020, background): “Typically, they either use text embeddings to represent entities/relations and align heterogeneous representations into the same space [37, 39, 43], or employ largescale raw corpora containing co-occurrence of entities to enrich the graph embeddings [42].”