Injecting Logical Background Knowledge into Embeddings for Relation Extraction

By Tim Rocktaschel, Sameer Singh, Sebastian Riedel. 2015

278 citations recorded by the source corpus.

Matrix factorization approaches to relation extraction provide several attractive features: they support distant supervision, handle open schemas, and leverage unlabeled data. Unfortunately, these methods share a shortcoming with all other distantly supervised approaches: they cannot learn to extract target relations without existing data in the knowledge base, and likewise, these models are inaccurate for relations with sparse data. Rule-based extractors, on the other hand, can be easily extended to novel relations and improved for existing but inaccurate relations, through first-order formulae that capture auxiliary domain knowledge. However, usually a large set of such formulae is necessary to achieve generalization. In this paper, we introduce a paradigm for learning low-dimensional embeddings of entity-pairs and relations that combine the advantages of matrix factorization with first-order logic domain knowledge. We introduce simple approaches for estimating such embeddings, as well as a novel training algorithm to jointly optimize over factual and first-order logic information. Our results show that this method is able to learn accurate extractors with little or no distant supervision alignments, while at the same time generalizing to textual patterns that do not appear in the formulae.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Long Short-Term Memory-Networks for Machine Reading (2016, background): “More recently, a few approaches have used low dimensional embeddings of entities and relations to enhance symbolic representations such as first-order logic (Rocktäschel et al., 2015; Bordes et al., 2011).”
Benefits of Intermediate Annotations in Reading Comprehension (2020, background): “Our auxiliary training with intermediate annotations is inspired by extensive related work on training models using side information or domain knowledge beyond labels (Mann and McCallum, 2008; Chang et al., 2007; Ganchev et al., 2010; Rocktaschel et al., 2015).”
RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems (2018, methodology): “ diag(r). The scoring function is hence defined as fr(h,t)= h⊤diag(r)t = ˝ d i=1 [r]i ·[h]i ·[t]i. Researchers also propose incorporating auxiliary information, such as entity types [39], logic rules [21], and textual descriptions [46] to assist KGE. However, these methods are more suitable for in-g”
DeepProbLog: Neural Probabilistic Logic Programming (2018, methodology): “[33] inject background knowledge into a matrix factorization model for relation extraction, by adding differentiable loss terms for propositionalized first-order rules.”
Logic Tensor Networks for Semantic Image Interpretation (2017, background): “The area of Statistical Relational Learning (SRL), or statistical AI (StarAI), seeks to combine learning from data with symbolic knowledge in the presence of uncertainty [Wang and Domingos, 2008; Bröcheler et al., 2012; Gutmann et al., 2010; Diligenti et al., 2015; Rocktaschel et al., 2015; Ravkic e”
Model-Agnostic Interpretability of Machine Learning (2016, methodology): “Using a white box makes it easier to incorporate user feedback in systems like iBCM (Kim et al., 2015), or injecting logic into matrix factorization (Rocktaschel et al., 2015).”
Modeling Relational Data with Graph Convolutional Networks (2017, background): “Beyond modeling knowledge bases, R-GCNs can be generalized to other applications where relation factorization models have been shown effective (e.g. relation extraction).”
NLProlog: Reasoning with Weak Unification for Question Answering in Natural Language (2019, background): “Moreover, such models tend to require large amounts of training data to generalise correctly, and incorporating background knowledge is still an open problem (Rocktäschel et al., 2015; Weissenborn et al., 2017a; Rocktäschel and Riedel, 2017; Evans and Grefenstette, 2017).”
Embedding Logical Queries on Knowledge Graphs (2018, background): “However, existing work in this area primarily focuses on using logical reasoning to improve edge prediction in knowledge graphs [14, 13, 27], for example, by using logical rules as regularization [15, 20, 35, 37].”
Transforming Dependency Structures to Logical Forms for Semantic Parsing (2016, background): “, 2015) and relation extraction (Rocktäschel et al., 2015).”
Knowledge Graph Completion via Complex Tensor Factorization (2017, methodology): “Injecting prior knowledge in the form of Horn clauses in the objective loss of the Universal Schema model has also been considered (Rocktaschel et al., 2015).”
Knowledge Graph Embedding: A Survey of Approaches and Applications (2017, background): “, entity types [25], [26], relation paths [27], [28], [29], textual descriptions [30], [31], [32], [33], and even logical rules [23], [34], [35], to learn more pre-”
Multi-Task Feature Learning for Knowledge Graph Enhanced Recommendation (2019, methodology): “t semantics of entities and relations, such as RESCAL [20], ANALOGY [19], and HolE [14]. Recently, researchers also propose incorporating auxiliary information, such as entity types [36], logic rules [24], and textual descriptions [46] to assist KGE. The above KGE methods can also be incorporated in”
Relation Extraction with Multi-instance Multi-label Convolutional Neural Networks (2016, background): “Further effects are also made to model missing data (Ritter et al., 2013), reduce noise (Roth et al., 2013), inject logical background knowledge (Rockt¨aschel et al., 2015), etc.”
Adversarial Sets for Regularising Neural Link Predictors (2017, methodology): “While the algorithm above is much more efﬁcient than grounding out the clauses over the entity space (as in Rocktäschel et al. [2015]), it still requires the inner loop of maximising the inconsistency loss.”