Hyperbolic Attention Networks

By Caglar Gulcehre, Misha Denil, Mateusz Malinowski, Ali Razavi, Razvan Pascanu, Karl Moritz Hermann, Peter W. Battaglia, Victor Bapst, David Raposo, Adam Santoro and 1 more. 2018

276 citations recorded by the source corpus.

We introduce hyperbolic attention networks to endow neural networks with enough capacity to match the complexity of data with hierarchical and power-law structure. A few recent approaches have successfully demonstrated the benefits of imposing hyperbolic geometry on the parameters of shallow networks. We extend this line of work by imposing hyperbolic geometry on the activations of neural networks. This allows us to exploit hyperbolic geometry to reason about embeddings produced by deep networks. We achieve this by re-expressing the ubiquitous mechanism of soft attention in terms of operations defined for hyperboloid and Klein models. Our method shows improvements in terms of generalization on neural machine translation, learning on graphs and visual question answering tasks while keeping the neural representations compact.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

Universal Transformers (2018, methodology): “This is partly similar to the Recursive Transformer, which ties the weights of its self-attention layers across depth (Gulcehre et al., 2018) 5 .”
Relational inductive biases, deep learning, and graph networks (2018, background): “…of deep learning and structured approaches, which focuses on approaches for reasoning about explicitly structured data, in particular graphs (e.g. Scarselli et al., 2009b; Bronstein et al., 2017; Gilmer et al., 2017; Wang et al., 2018c; Li et al., 2018; Kipf et al., 2018; Gulcehre et al., 2018).”
Learning Visual Question Answering by Bootstrapping Hard Attention (2018, background): “However, only soft attention is used in the majority of Visual QA works [7,8,9,10,11,12,43,44,45,46,47,48,49,50,51,52].”
Differentiating through the Fréchet Mean (2020, methodology): “, 2019) and the Einstein midpoint (Gulcehre et al., 2019) methods.”
Hyperbolic Image Embeddings (2019, background): “Another direction of research, more relevant to the present work, is based on imposing hyperbolic structure on activations of neural networks [11, 12].”
Hyperbolic Graph Convolutional Neural Networks (2019, unknown): “Hyperbolic geometry has been applied to neural networks, to problems of computer vision or natural language processing [8, 14, 18, 38].”
Multi-relational Poincaré Graph Embeddings (2019, methodology): “Hyperbolic analogues of several other algorithms have been developed since, such as Poincaré GloVe [32] and Hyperbolic Attention Networks [16].”
Novel positional encodings to enable tree-based transformers (2019, unknown): “Rather than explicitly modeling hierarchically structured data, some recent work imposes hyperbolic geometry on the activations of neural networks (Gulcehre et al., 2018).”
Learning Distributed Representations of Symbolic Structure Using Binding and Unbinding Operations (2018, background): “fforts on learning structured distributed representations can be categorised into two types: enforcing a strong global geometrical constraint on the representation space, such as hyperbolic embedding [21, 42], and introducing inductive biases into the architecture of networks, including Relational M”
Hyperbolic Graph Neural Networks (2019, background): “[21] proposed hyperbolic neural networks and hyperbolic attention networks, respectively, with the aim of extending deep learning methods to hyperbolic space.”