Lifted Rule Injection for Relation Embeddings

By Thomas Demeester, Tim Rocktaschel, Sebastian Riedel. 2016

152 citations recorded by the source corpus.

Methods based on representation learning currently hold the state-of-the-art in many natural language processing and knowledge base inference tasks. Yet, a major challenge is how to efficiently incorporate commonsense knowledge into such models. A recent approach regularizes relation and entity representations by propositionalization of first-order logic rules. However, propositionalization does not scale beyond domains with only few entities and rules. In this paper we present a highly efficient method for incorporating implication rules into distributed representations for automated knowledge base construction. We map entity-tuple embeddings into an approximately Boolean space and encourage a partial ordering over relation embeddings based on implication rules mined from WordNet. Surprisingly, we find that the strong restriction of the entity-tuple embedding space does not hurt the expressiveness of the model and even acts as a regularizer that improves generalization. By incorporating few commonsense rules, we achieve an increase of 2 percentage points mean average precision over a matrix factorization baseline, while observing a negligible increase in runtime.

Subfields: Artificial Intelligence, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Graph Learning

Citations in the corpus

Zero-Shot Relation Extraction via Reading Comprehension (2017, background): “Rockt¨aschel et al. (2015) and Demeester et al. (2016) use inference rules to predict hidden knowledge-base relations from observed natural-language relations.”
DeepProbLog: Neural Probabilistic Logic Programming (2018, background): “[33, 34, 35, 36, 37, 38] all center around including logical background knowledge as a regularizer during training.”
End-to-end Differentiable Proving (2017, background): “Another line of work [45–49] regularizes distributed representations via domain-speciﬁc rules, but does not learn such rules from data and only supports a restricted subset of ﬁrst-order logic.”
Meta-Learning Improves Lifelong Relation Extraction (2019, methodology): “Zero-shot relation extraction approaches (Rockt ¨ aschel et al., 2015; Demeester et al., 2016; Levy et al., 2017; Oba-muyide and Vlachos, 2018) can extract unseen relations, but at lower performance levels, and are unable to continually exploit newly available supervision to improve performance…”
Learning Relational Representations with Auto-encoding Logic Programs (2019, background): “Several works [Minervini et al. , 2017; Demeester et al. , 2016] impose logical constraints on embed-dings but do not retain the relational representation.”
SimplE Embedding for Link Prediction in Knowledge Graphs (2018, methodology): “It has been observed in several works that hybrid models (models based on a combination of the approaches in these categories) may result in higher predictive performance compared to models based on approaches in a single category (Lin et al., 2015a; Neelakantan et al., 2015; Demeester et al., 2016;”
Poincaré Embeddings for Learning Hierarchical Representations (2017, background): “[10] showed that including prior information about hypernymy relations in form of logical rules can improve the quality of word embeddings.”
Knowledge Graph Embedding: A Survey of Approaches and Applications (2017, methodology): “[111] recently proposed an extension of [35], the key idea of which is to impose logical implications by regularizing relation embeddings so as to avoid grounding.”
Exploring End-to-End Differentiable Natural Logic Modeling (2020, methodology): “One uses symbolic constraints as regularizers to equip neural models with the corresponding inductive bias (Demeester et al., 2016; Diligenti et al., 2017; Donadello et al., 2017; Xu et al., 2018; Li and Srikumar, 2019).”
Lifted Relational Neural Networks: Efficient Learning of Latent Relational Structures (2018, methodology): “Recently 3 two other frameworks for combining logic programming and neural networks have been introduced (Rockt¨aschel & Riedel, 2016; Cohen, 2016).”
Knowledge Graphs (2020, unknown): “Finally, in the intersection of deductive and inductive knowledge, we mention the topics of entailment-aware knowledge graph embeddings [125, 207], that incorporate rules and/or ontologies when computing plausibility; expressive graph neural networks proven capable of complex classification analogou”
Adversarial Sets for Regularising Neural Link Predictors (2017, methodology): “Given an assumption A expressed as a FOL clause BODY ⇒ HEAD , as in Eq.”
Embedding Logical Queries on Knowledge Graphs (2018, background): “However, existing work in this area primarily focuses on using logical reasoning to improve edge prediction in knowledge graphs [14, 13, 27], for example, by using logical rules as regularization [15, 20, 35, 37].”
T-REx: A Large Scale Alignment of Natural Language with Knowledge Base Triples (2018, methodology): “This can be alleviated through incorporating implication rules in the alignment process [Demeester et al.2016].”
Neuro-Symbolic = Neural + Logical + Probabilistic (2019, methodology): “, 2017] FOL Lifted Rule Injection [Demeester et al., 2016] X Implication rules Adverserial Set Regularisation [Minervini et al.”