Lie Access Neural Turing Machine
By Greg Yang. 2016
17 citations recorded by the source corpus.
Following the recent trend in explicit neural memory structures, we present a new design of an external memory, wherein memories are stored in an Euclidean key space $\mathbb R^n$. An LSTM controller performs read and write via specialized read and write heads. It can move a head by either providing a new address in the key space (aka random access) or moving from its previous position via a Lie group action (aka Lie access). In this way, the "L" and "R" instructions of a traditional Turing Machine are generalized to arbitrary elements of a fixed Lie group action. For this reason, we name this new model the Lie Access Neural Turing Machine, or LANTM. We tested two different configurations of LANTM against an LSTM baseline in several basic experiments. We found the right configuration of LANTM to outperform the baseline in all of our experiments. In particular, we trained LANTM on addition of $k$-digit numbers for $2 \le k \le 16$, but it was able to generalize almost perfectly to $17 \le k \le 32$, all with the number of parameters 2 orders of magnitude below the LSTM baseline.
Subfields: Artificial Intelligence, Electrical and Electronic Engineering
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Kyunghyun Cho
- Mohit Bansal
- Sergey Levine
- Quoc V. Le
- Jitendra Malik
- Jianfeng Gao
- Ruslan Salakhutdinov
In Electrical and Electronic Engineering
- Yoshua Bengio
- Kyunghyun Cho
- Jitendra Malik
- A. P. Sarath Chandar
- Caglar Gulcehre
- Ke Li
- Anirudh Goyal
- Joelle Pineau
Citations in the corpus
- Learning to Optimize (2016, background): “Subsequent work proposes variants of this model that use different primitive memory access operations [14], more expressive operations [16, 28] or other non-differentiable operations [30, 29].”
- Dynamic Neural Turing Machine with Continuous and Discrete Addressing Schemes (2018, background): “In parallel to our work, Yang (2016) and Graves et al. (2016) proposed new memory access mechanisms to improve NTM type of models.”