Learning Structured Embeddings of Knowledge Bases

By Antoine Bordes, Jason Aaron Edward Weston, Ronan Collobert, Yoshua Bengio. 2011

940 citations recorded by the source corpus.

Many Knowledge Bases (KBs) are now readily available and encompass colossal quantities of information thanks to either a long-term funding effort (e.g. WordNet, OpenCyc) or a collaborative process (e.g. Freebase, DBpedia). However, each of them is based on a different rigorous symbolic framework which makes it hard to use their data in other systems. It is unfortunate because such rich structured knowledge might lead to a huge leap forward in many other areas of AI like nat- ural language processing (word-sense disambiguation, natural language understanding, ...), vision (scene classification, image semantic annotation, ...) or collaborative filtering. In this paper, we present a learning process based on an innovative neural network architecture designed to embed any of these symbolic representations into a more flexible continuous vector space in which the original knowledge is kept and enhanced. These learnt embeddings would allow data from any KB to be easily used in recent machine learning meth- ods for prediction and information retrieval. We illustrate our method on WordNet and Freebase and also present a way to adapt it to knowledge extraction from raw text.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Graph Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Graph Learning

Citations in the corpus

Long Short-Term Memory-Networks for Machine Reading (2016, background): “More recently, a few approaches have used low dimensional embeddings of entities and relations to enhance symbolic representations such as first-order logic (Rocktäschel et al., 2015; Bordes et al., 2011).”
Order-Embeddings of Images and Language (2015, methodology): “Bordes et al. (2011) and Socher et al. (2013) also evaluate on the WordNet hierarchy, but they use other relations in WordNet as training data (and external linguistic data, in Socher’s case).”
Knowledge Graph Embedding by Translating on Hyperplanes (2014, methodology): “Distant Model (Bordes et al. 2011) introduces two independent projections to the entities in a relation.”
DKN: Deep Knowledge-Aware Network for News Recommendation (2018, methodology): “In addition to translation-based methods [4, 18, 26, 48] used in DKN, researchers have also proposed many other models such as Structured Embedding [5], Latent Factor Model [17], Neural Tensor Network [37] and GraphGAN [42].”
Learning Entity and Relation Embeddings for Knowledge Graph Completion (2015, methodology): “set up in (Bordes et al. 2011; 2013), we conduct experiments using the data sets WN18 and FB15K.”
Neural-Symbolic Computing: An Effective Methodology for Principled Integration of Machine Learning and Reasoning (2019, background): “[4], [41], NTN[47], [48], etc.”
Deep Learning: Methods and Applications (2014, background): “An early work in this space came from [37], where a process is introduced to automatically learn structured distributed embeddings of knowledge bases.”
Connecting Language and Knowledge Bases with Embedding Models for Relation Extraction (2013, methodology): “in (Nickel et al., 2011; Bordes et al., 2011; Bordes et al., 2012)): we chose in this work to use the approach of (Bordes et al.”
Irreflexive and Hierarchical Relations as Translations (2013, methodology): “We also intend to consider learning translations of word embedding, either from free text as in (Mikolov et al., 2013) or from ( subject, verb, object ) triplets as in (Bordes et al., 2011).”
A semantic matching energy function for learning with multi-relational data (2013, background): “A relation is denoted by a triplet ( lhs , rel , rhs ), where lhs is the left entity, rhs the right one and rel the type of relation between them.”
A Comprehensive Survey of Graph Embedding: Problems, Techniques, and Applications (2017, background): “, [90], [91], [92]) learns embedding for both nodes and edges.”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, methodology): “This has been exploited successfully to design image search systems in which images and queries are mapped to the same semantic space (Weston et al. , 2011).”
From machine learning to machine reasoning (2011, background): “This system reaches near state-of-the-art performance while running hundreds of times faster than natural language processing systems with comparable performance.”
Embedding Entities and Relations for Learning and Inference in Knowledge Bases (2014, methodology): “Models B r A T r Scoring Function Distance (Bordes et al., 2011) - Comparisons among several multi-relational models in their scoring functions.”
A Review of Relational Machine Learning for Knowledge Graphs (2015, methodology): “The structured embedding (SE) model [93] extends this idea to multirelational data by modeling the score of a triple xijk as”
NaturalLI: Natural Logic Inference for Common Sense Reasoning (2014, methodology): “Chen et al. (2013) and Socher et al. (2013) use Neural Tensor Networks to predict unseen relation triples in WordNet and Freebase, following a line of work by Bordes et al. (2011) and Jenatton et al. (2012).”
StarSpace: Embed All The Things! (2017, methodology): “TransE was shown to out-perform RESCAL (Nickel, Tresp, and Kriegel 2011), RFM (Jenatton et al. 2012), SE (Bordes et al. 2011) and SME (Bordes et al. 2014) and is considered a standard benchmark method.”
Knowledge Graph Embedding via Dynamic Mapping Matrix (2015, methodology): “2012; 2014) O(Nem) O(Nt) SE (Bordes et al. 2011) O(Nem + 2Nrn(2))(m = n) O(2m(2)Nt) SME(linear) (Bordes et al.”