Convolutional 2D Knowledge Graph Embeddings
By Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel. 2017
3,143 citations recorded by the source corpus.
Link prediction for knowledge graphs is the task of predicting missing relationships between entities. Previous work on link prediction has focused on shallow, fast models which can scale to large knowledge graphs. However, these models learn less expressive features than deep, multi-layer models — which potentially limits performance. In this work we introduce ConvE, a multi-layer convolutional network model for link prediction, and report state-of-the-art results for several established datasets. We also show that the model is highly parameter efficient, yielding the same performance as DistMult and R-GCN with 8x and 17x fewer parameters. Analysis of our model suggests that it is particularly effective at modelling nodes with high indegree — which are common in highly-connected, complex knowledge graphs such as Freebase and YAGO3. In addition, it has been noted that the WN18 and FB15k datasets suffer from test set leakage, due to inverse relations from the training set being present in the test set — however, the extent of this issue has so far not been quantified. We find this problem to be severe: a simple rule-based model can achieve state-of-the-art results on both WN18 and FB15k. To ensure that models are evaluated on datasets where simply exploiting inverse relations cannot yield competitive results, we investigate and validate several commonly used datasets — deriving robust variants where necessary. We then perform experiments on these robust datasets for our own and several previously proposed models, and find that ConvE achieves state-of-the-art Mean Reciprocal Rank across all datasets.
Subfields: Artificial Intelligence, Management Science and Operations Research, Graph Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Percy Liang
- Luke S. Zettlemoyer
- Yoshua Bengio
- Graham Neubig
- Yejin Choi
- Noah A. Smith
In Management Science and Operations Research
- Wei Hu
- Zequn Sun
- William L. Hamilton
- Partha P. Talukdar
- Soumya Sanyal
- Shikhar Vashishth
- Vikram Nitin
- Bilal Abu-Salih
In Graph Learning
- William L. Hamilton
- Jure Leskovec
- Jian Tang
- Partha P. Talukdar
- Shirui Pan
- Philip S. Yu
- Xiang Ren
- Thomas Kipf
Citations in the corpus
- Knowledge Enhanced Contextual Word Representations (2019, methodology): “…distance models (Bordes et al., 2013; Wang et al., 2014b; Lin et al., 2015; Xiao et al., 2016) which use a distance-based scoring function, and linear models (Nickel et al., 2011; Yang et al., 2014; Trouillon et al., 2016; Dettmers et al., 2018) which use a similarity-based scoring function.”
- Commonsense Knowledge Mining from Pretrained Models (2019, background): “This problem of train-test leakage is typical in knowledge base completion tasks (Toutanova et al., 2015; Dettmers et al., 2018).”
- Recurrent Event Network for Reasoning over Temporal Knowledge Graphs (2019, methodology): “There are methods which embed entities and relations into low-dimensional spaces [3, 27, 24, 7].”
- Modeling Relational Data with Graph Convolutional Networks (2017, methodology): “For example, the graph autoencoder model could be considered in combination with other factorization models, such as ConvE [34], which can be better suited for modeling asymmetric relations.”
- Differentiable Reasoning over a Virtual Knowledge Base (2020, background): “Knowledge Graph embeddings (Bordes et al., 2013; Yang et al., 2014; Dettmers et al., 2018) attach continuous representations to discrete symbols which allow them to be incorporated in deep networks (Yang & Mitchell, 2017).”
- Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge (2020, background): “More recently, the past decade has seen huge amount of work on knowledge base embeddings (Bordes et al., 2013; Lin et al., 2015; Trouillon et al., 2017; Dettmers et al., 2018) which enable generalization through similarities between”
- Knowledge Graph Embedding for Link Prediction (2020, methodology): “In FB15K-237, all models display globally worse performance; nonetheless, interestingly most of them manage to keep good performance on reexive relations, the exceptions being ANALOGY , SimplE [27], ConvE [11] and RSN [19].”
- Language Models as Knowledge Bases: On Entity Representations, Storage Capacity, and Paraphrased Queries (2020, background): “, 2016), ConvE (Dettmers et al., 2018) Graph neural networks nodes, node features, edges node embeddings DeepWalk (Perozzi et al.”
- KG-BERT: BERT for Knowledge Graph Completion (2019, background): “(cid:104) (cid:105) Recently, convolutional neural networks also show promising results for knowledge graph completion (Dettmers et al. 2018; Nguyen et al. 2018a; Schlichtkrull et al. 2018).”
- Domain-specific Knowledge Graphs: A survey (2020, methodology): “Example of KGE models include, but not limited to; Translating Embedding (TransE) [35], DistMult [36], Complex Embeddings (ComplEx) [37], Holographic Embeddings(HolE) [38], Convolutional 2D KG Embeddings (ConvE) [39], Convolution-based model (ConvKB) [40], etc.”
- Bootstrapping Entity Alignment with Knowledge Graph Embedding (2018, background): “Also, there exist quite a number of non-translational models for KG embedding, such as [Nickel et al. , 2011; Socher et al. , 2013; Yang et al. , 2017; Dettmers et al. , 2018].”
- CoLAKE: Contextualized Language and Knowledge Embedding (2020, methodology): “If the masked entity is not an anchor node, the MLM objective is similar to that used in semantic matching-based KE methods such as ConvE (Dettmers et al., 2018) and CoKE (Wang et al., 2019b), which enables CoLAKE to learn a large number of entity embeddings.”
- Temporal Knowledge Graph Reasoning Based on Evolutional Representation Learning (2021, methodology): “DistMult [41], ComplEx [35], R-GCN [28], ConvE [6], ConvTransE [30], RotatE [32] are selected as static models.”
- Quaternion Knowledge Graph Embeddings (2019, background): “More recently, convolution neural networks [Dettmers et al., 2018], graph convolutional networks [Schlichtkrull et al., 2018], and deep memory networks [Wang et al., 2018] also show promising performance on this task.”
- Adaptive Convolution for Multi-Relational Learning (2019, methodology): “Using simple models allows one to easily handle large-scale KBs, but usually at the cost of less model expressiveness (Dettmers et al., 2018).”
- GMNN: Graph Markov Neural Networks (2019, neutral): “Copyright 2019 by the author(s). classification (Dettmers et al., 2018).”
- Recurrent Event Network: Autoregressive Structure Inference over Temporal Knowledge Graphs (2019, methodology): “We use the codes for ConvE 8 and use implementation by Deep Graph Library 9 .”
- InteractE: Improving Convolution-based Knowledge Graph Embeddings by Increasing Feature Interactions (2019, methodology): “Simliar trend has been observed in (Dettmers et al. 2018; Sun et al. 2019a).”
- KBGAN: Adversarial Learning for Knowledge Graph Embeddings (2017, methodology): “Likewise, WN18RR is a subset of WN18 introduced by (Dettmers et al., 2017) which removes reversing relations and dramatically increases the difficulty of reasoning.”