Representing Text for Joint Embedding of Text and Knowledge Bases

By Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoifung Poon, Pallavi Choudhury, Michael Gamon. 2015

822 citations recorded by the source corpus.

Models that learn to represent textual and knowledge base relations in the same continuous latent space are able to perform joint inferences among the two kinds of relations and obtain high accuracy on knowledge base completion (Riedel et al., 2013). In this paper we propose a model that captures the compositional structure of textual relations, and jointly optimizes entity, knowledge base, and textual relation representations. The proposed model significantly improves performance over a model that does not share parameters among textual relations with common sub-structure.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Zero-Shot Relation Extraction via Reading Comprehension (2017, background): “Toutanova et al. (2015) proposed a similar approach that decomposes natural-language relations and computes their similarity in a universal schema setting; however, they did not extend their method to knowledge-base relations, nor did they attempt to recover out-of-schema relations as we do.”
ERNIE: Enhanced Language Representation with Informative Entities (2019, background): “In fact, some work has attempted to joint representation learning of words and entities for effectively leveraging external KGs and achieved promising results (Wang et al., 2014; Toutanova et al., 2015; Han et al., 2016; Yamada et al., 2016; Cao et al., 2017, 2018).”
Commonsense Knowledge Mining from Pretrained Models (2019, background): “This problem of train-test leakage is typical in knowledge base completion tasks (Toutanova et al., 2015; Dettmers et al., 2018).”
Compositional Fairness Constraints for Graph Embeddings (2019, background): “Freebase 15k-237 is a standard benchmark used for knowledge base completion (Toutanova et al., 2015).”
Matching the Blanks: Distributional Similarity for Relation Learning (2019, background): “resentation that has been extended to arbitrary textual input (Toutanova et al., 2015), and arbitrary entity pairs (Verga and McCallum, 2016).”
Commonsense Knowledge Base Completion (2016, methodology): “…2013; Gardner et al., 2014; West et al., 2014), particularly methods based on distributed representations and neural networks (Socher et al., 2013; Bordes et al., 2013; Bordes et al., 2014a; Bordes et al., 2014b; Yang et al., 2014; Neelakantan et al., 2015; Gu et al., 2015; Toutanova et al., 2015).”
A Neural Knowledge Language Model (2016, methodology): “The idea of jointly using Wikipedia and knowledge graphs has also been used in the context of enriching word embedding [37, 11, 25].”
KEPLER: A Unified Model for Knowledge Embedding and Pre-trained Language Representation (2019, background): “…and dialogue system (Madotto et al., 2018), but also some early works use text as additional information (Xie et al., 2016; An et al., 2018) or jointly train the knowledge and text embedding in the same space (Wang et al., 2014; Toutanova et al., 2015; Han et al., 2016; Cao et al., 2017, 2018).”
K-BERT: Enabling Language Representation with Knowledge Graph (2019, background): “Toutanova et al. (2015) proposed a model that captures the compositional structure of textual relations, and optimize entity, knowledge base, and textual relation representations in a joint manner.”
Convolutional 2D Knowledge Graph Embeddings (2017, neutral): “@1 DistMult [34] For FB15k-237, we could not replicate the basic model results from Toutanova et al. [32], where the models in general have better performance than what we can achieve.”
DAG-GNN: DAG Structure Learning with Graph Neural Networks (2019, methodology): “We construct the data set from triples in FB15K-237 (Toutanova et al., 2015), which is a subset of FreeBase with approximately 15k entities and 237 relations.”
Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text (2018, background): “Another line of work has looked at learning combined representations of KBs and text for relation extraction and Knowledge Base Completion (KBC) (Lao et al., 2012; Riedel et al., 2013; Toutanova et al., 2015; Verga et al., 2016; Das et al., 2017b; Han et al., 2016).”
Learning Cross-Context Entity Representations from Text (2020, background): “…entities by building knowledge base embeddings (Bordes et al., 2011; Socher et al., 2013; Yang et al., 2014; Toutanova et al., 2016; Vilnis et al., 2018), and by jointly embedding knowledge bases and information from textual mentions (Riedel et al., 2013; Toutanova et al., 2015; Hu et al., 2015).”
GAKE: Graph Aware Knowledge Embedding (2016, background): “There are some other works incorporate additional information, such as text(Toutanova and Chen, 2015; Toutanova et al., 2015) and entity types(Guo et al., 2015).”
Compositional Learning of Embeddings for Relation Paths in Knowledge Base and Text (2016, background): “The problem is worsened when the input is augmented with unlabeled text, which has been shown to improve performance (Lao et al., 2012; Gard-ner et al., 2013; Riedel et al., 2013; Gardner et al., 2014; Toutanova and Chen, 2015).”
Multi-Hop Knowledge Graph Reasoning with Reward Shaping (2018, neutral): “On much larger KGs (FB15k-237, WN18RR, and NELL-995), we did not observe a signiﬁcant slowdown before severe overﬁtting occurs and the dev set performance starts to drop.”