\infty-former: Infinite Memory Transformer

By Pedro Henrique Martins, Zita Marinho, André F. T. Martins. 2021

10 citations recorded by the source corpus.

Transformers are unable to model long-term memories effectively, since the amount of computation they need to perform grows with the context length. While variations of efficient transformers have been proposed, they all have a finite memory capacity and are forced to drop old information. In this paper, we propose the \infty-former, which extends the vanilla transformer with an unbounded long-term memory. By making use of a continuous-space attention mechanism to attend over the long-term memory, the \infty-former’s attention complexity becomes independent of the context length, trading off memory length with precision.In order to control where precision is more important, \infty-former maintains “sticky memories,” being able to model arbitrarily long contexts while keeping the computation budget fixed.Experiments on a synthetic sorting task, language modeling, and document grounded dialogue generation demonstrate the \infty-former’s ability to retain information from long sequences.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jiashi Feng
2. Xinchao Wang
3. Weihao Yu
4. Shuicheng Yan
5. Pan Zhou
6. Yichen Zhou
7. Amirkeivan Mohtashami
8. Romy Mi Luo

In Computer Vision and Pattern Recognition

1. Jiashi Feng
2. Xinchao Wang
3. Shuicheng Yan
4. Weihao Yu
5. Dacheng Tao
6. Pan Zhou
7. Luc Van Gool
8. Yichen Zhou

In Model Architectures

1. Jiashi Feng
2. Shuicheng Yan
3. Weihao Yu
4. Xinchao Wang
5. Pan Zhou
6. Amirkeivan Mohtashami
7. Yichen Zhou
8. Romy Mi Luo

Citations in the corpus

• Landmark Attention: Random-Access Infinite Context Length for Transformers (2023, methodology): “Infinite Memory Transformers [22] map the input to a continuous space and then sample points to be used for memory in the next step according to the probability distribution defined by the attention mechanism.”
• MetaFormer is Actually What You Need for Vision (2021, background): “Some recent approaches [31, 38, 39, 44] explore other types of token mixers within the MetaFormer architecture, and have demonstrated encouraging performance.”