Embedding Entities and Relations for Learning and Inference in Knowledge Bases
By Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, Li Deng. 2014
3,841 citations recorded by the source corpus.
Abstract: We consider learning representations of entities and relations in KBs using the neural-embedding approach. We show that most existing models, including NTN (Socher et al., 2013) and TransE (Bordes et al., 2013b), can be generalized under a unified learning framework, where entities are low-dimensional vectors learned from a neural network and relations are bilinear and/or linear mapping functions. Under this framework, we compare a variety of embedding models on the link prediction task. We show that a simple bilinear formulation achieves new state-of-the-art results for the task (achieving a top-10 accuracy of 73.2% vs. 54.7% by TransE on Freebase). Furthermore, we introduce a novel approach that utilizes the learned relation embeddings to mine logical rules such as "BornInCity(a,b) and CityInCountry(b,c) => Nationality(a,c)". We find that embeddings learned from the bilinear objective are particularly good at capturing relational semantics and that the composition of relations is characterized by matrix multiplication. More interestingly, we demonstrate that our embedding-based rule extraction approach successfully outperforms a state-of-the-art confidence-based rule mining approach in mining Horn rules that involve compositional reasoning.
Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Graph Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Percy Liang
- Jianfeng Gao
- Luke S. Zettlemoyer
- Yoshua Bengio
- Graham Neubig
- Noah A. Smith
- Yejin Choi
In Representation & Self-Supervised Learning
- William L. Hamilton
- Percy Liang
- Jure Leskovec
- Zhiyuan Liu
- Yoshua Bengio
- Richard Socher
- Jianfeng Gao
- Xipeng Qiu
In Graph Learning
- Jure Leskovec
- William L. Hamilton
- Jian Tang
- Marinka Zitnik
- Seyed Mehran Kazemi
- Partha P. Talukdar
- Sebastian Riedel
- Stephan Günnemann
Citations in the corpus
- Knowledge Enhanced Contextual Word Representations (2019, methodology): “…distance models (Bordes et al., 2013; Wang et al., 2014b; Lin et al., 2015; Xiao et al., 2016) which use a distance-based scoring function, and linear models (Nickel et al., 2011; Yang et al., 2014; Trouillon et al., 2016; Dettmers et al., 2018) which use a similarity-based scoring function.”
- Neural Approaches to Conversational AI (2018, positive): “The use of vectors rather than matrices for relation representations is inspired by the bilinear-diag model (Yang et al., 2015), which restricts the relation representations to the class of diagonal matrices.”
- RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems (2018, methodology): “ h, r and t. Therefore, TransE assumes the score function fr(h,t)= ∥h+r−t∥2 2 is low if (h,r,t)holds, and high otherwise. (2) Semantic matching models, such as ANALOGY [19], ComplEx [28], and DisMult [41], measure plausibility of knowledge triples by matching latent semantics of entities and relatio”
- Compositional Fairness Constraints for Graph Embeddings (2019, background): “, [5, 20, 22]) are also amenable to our framework.”
- On Multiplicative Integration with Recurrent Neural Networks (2016, background): “[28] explored neural embedding approaches in knowledge bases by formulating relations as bilinear and/or linear mapping functions, and compared a variety of embedding models on the link prediction task.”
- Complex Embeddings for Simple Link Prediction (2016, methodology): “The original multi-linear DistMult model is symmetric in subject and object for every relation (Yang et al., 2015) and achieves good performance, presumably due to its simplicity.”
- Commonsense Knowledge Base Completion (2016, methodology): “We first consider bilinear models, since they have been found useful for KBC in past work (Nickel et al., 2011; Jenatton et al., 2012; Garc´ıa-Dur´an et al., 2014; Yang et al., 2014).”
- Improving Information Extraction by Acquiring External Evidence with Reinforcement Learning (2016, methodology): “Recent work has explored several techniques to perform Knowledge Base Completion (KBC) such as vector space models and graph traversal (Socher et al., 2013; Yang et al., 2014; Gardner et al., 2014; Neelakantan et al., 2015; Guu et al., 2015).”
- A Review of Relational Machine Learning for Knowledge Graphs (2015, methodology): “As noted in [95], under unit-norm constraints on ei; ej and using the squared Euclidean distance, we can rewrite (16) as follows:”
- Representing Text for Joint Embedding of Text and Knowledge Bases (2015, methodology): “We also limit the candidate entities to ones that have types consistent with the position in the relation triple (Chang et al., 2014; Yang et al., 2015), where the types are approximated following Toutanova and Chen (2015).”
- Differentiable Reasoning over a Virtual Knowledge Base (2020, background): “Knowledge Graph embeddings (Bordes et al., 2013; Yang et al., 2014; Dettmers et al., 2018) attach continuous representations to discrete symbols which allow them to be incorporated in deep networks (Yang & Mitchell, 2017).”
- Traversing Knowledge Graphs in Vector Space (2015, methodology): “The Bilinear-Diag model of Yang et al. (2015) is a special case of the bilinear model with the relation matrices constrained to be diagonal.”
- Knowledgeable Reader: Enhancing Cloze-Style Reading Comprehension with External Commonsense Knowledge (2018, methodology): “, 2015) using the BILINEAR (Yang et al., 2014) model.”
- How Does NLP Benefit Legal System: A Summary of Legal Artificial Intelligence (2020, background): “Many embedding methods have been proved effective (Mikolov et al., 2013; Joulin et al., 2016; Pennington et al., 2014; Peters et al., 2018; Yang et al., 2014; Bordes et al., 2013; Lin et al., 2015) and they are crucial for the effectiveness of the downstream tasks.”