Poincaré Embeddings for Learning Hierarchical Representations

By Maximilian Nickel, Douwe Kiela. 2017

1,718 citations recorded by the source corpus.

Representation learning has become an invaluable approach for learning from symbolic data such as text and graphs. However, while complex symbolic datasets often exhibit a latent hierarchical structure, state-of-the-art methods typically learn embeddings in Euclidean vector spaces, which do not account for this property. For this purpose, we introduce a new approach for learning hierarchical representations of symbolic data by embedding them into hyperbolic space -- or more precisely into an n-dimensional Poincare ball. Due to the underlying hyperbolic geometry, this allows us to learn parsimonious representations of symbolic data by simultaneously capturing hierarchy and similarity. We introduce an efficient algorithm to learn the embeddings based on Riemannian optimization and show experimentally that Poincare embeddings outperform Euclidean embeddings significantly on data with latent hierarchies, both in terms of representation capacity and in terms of generalization ability.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Computer Vision and Pattern Recognition

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “In this way learned object representations can also benefit from known inductive biases that focus on feature hierarchies, invariances, and spatio-temporal coherence (Becker and Hinton, 1992), sparsity (Olshausen and Field, 1996), or non-Euclidean feature spaces (Nickel and Kiela, 2017).”
Multi-Object Representation Learning with Iterative Variational Inference (2019, methodology): “This includes disentangled representations through the use of β -VAEs (Higgins et al., 2017a), adversarial autoencoders (Makhzani et al., 2015), Factor VAEs (Kim & Mnih, 2018), and improved generalization through non-euclidean embed-dings (Nickel & Kiela, 2017).”
Visualizing and Measuring the Geometry of BERT (2019, background): “The simplicity of Pythagorean tree embeddings, as well as the fact that they may be approximated by a simple random model, suggests they may be a generally useful alternative to approaches to tree embeddings that require hyperbolic geometry [18].”
Low-Dimensional Hyperbolic Knowledge Graph Embeddings (2020, background): “Recent research has proposed embedding hierarchical graphs into these spaces instead of conventional Euclidean space (Nickel and Kiela, 2017; Sala et al., 2018).”
Geom-GCN: Geometric Graph Convolutional Networks (2020, methodology): “We employ three embedding methods, Isomap (Tenenbaum et al., 2000), Poincare embedding (Nickel & Kiela, 2017), and struc2vec (Ribeiro et al., 2017), which result in three Geom-GCN variants: Geom-GCN-I, GeomGCN-P, and Geom-GCN-S.”
Hyperspherical Prototype Networks (2019, neutral): “Several works have investigated prior and semantic knowledge in hyperbolic spaces [8, 32, 40].”
Opportunities in Machine Learning for Healthcare (2018, neutral): “and domain knowledge into representation learning. There has also been some initial exploration into learning representations of other data types that simultaneously capture hierarchy and similarity (Nickel and Kiela, 2017), and recent work has examined using diagnostic code ontologies to aid predic”
Hyperbolic Neural Networks (2018, methodology): “Using an open source implementation 7 of [21], we pre-trained Poincaré embeddings of the Word-Net noun hierarchy (82,115 nodes).”
A Review of Challenges and Opportunities in Machine Learning for Health. (2018, background): “There has also been some initial exploration into learning representations of other data types that simultaneously capture hierarchy and similarity 110-112 .”
Hyperbolic Image Embeddings (2019, background): “Hyperbolic embeddings in the natural language processing field have recently been very successful [29, 30].”
Machine Learning on Graphs: A Model and Comprehensive Taxonomy (2020, background): “2 Poincaré [83] Lorentz [84] Product [49]”
Hyperbolic Graph Convolutional Neural Networks (2019, methodology): “For instance, Poincaré embeddings [29] capture the hyperbolic properties of real graphs by learning shallow embeddings with hyperbolic distance metric and Riemannian optimization.”
Multi-relational Poincaré Graph Embeddings (2019, methodology): “Nickel and Kiela [25] ﬁrst embedded the transitive closure 1 of the WordNet noun hierarchy, in the Poincaré ball, showing that low-dimensional hyperbolic embeddings can signiﬁcantly outperform higher-dimensional Euclidean embeddings in terms of both representation capacity and generalization…”
A Survey on Differential Privacy for Unstructured Data Content (2022, methodology): “For instance, Poincaré embeddings [72] employ a hierarchical semantic relationship between words to obtain generalized word vectors and their subsets.”
Differentiating through the Fréchet Mean (2020, methodology): “Recent advancements in geometric representation learning have utilized hyperbolic space for tree embedding tasks (Nickel & Kiela, 2017; 2018; Yu & De Sa, 2019).”
Hyperbolic Attention Networks (2018, background): “Following this exciting line of research, recently the machine learning community has gained interest in learning non-Euclidean embeddings directly from data [28, 8, 33, 29, 38, 6].”
Hyperspherical Variational Auto-Encoders (2018, background): “Another approach by Nickel and Kiela (2017), capitalizes on the hierarchical structure present in some data types.”