Augmenting Self-attention with Persistent Memory

By Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herv'e J'egou, Armand Joulin. 2019

160 citations recorded by the source corpus.

Transformer networks have lead to important progress in language modeling and machine translation. These models include two consecutive modules, a feed-forward layer and a self-attention layer. The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers. Building upon this intuition, we propose a new model that solely consists of attention layers. More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer. Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer. Our evaluation shows the benefits brought by our model on standard character and word level language modeling benchmarks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Reformer: The Efficient Transformer (2020, methodology): “In addition to standard methods such as precision reduction and gradient checkpointing (Sohoni et al., 2019), more efﬁcient versions of the Transformer model’s self-attention mechanism (Sukhbaatar et al., 2019a;b) have also recently been explored.”
Transformer Feed-Forward Layers Are Key-Value Memories (2020, methodology): “Sukhbaatar et al. (2019) make an analogous observation, and incorporate the parameters of the feed-forward layers as persistent memory cells in the self-attention layers.”
Efficient Transformers: A Survey (2020, background): “This can be thought of as a form of parameter attention (Sukhbaatar et al., 2019).”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, neutral): “This is in contrast with concurrent approaches in machine translation which have investigated the use of persistent memories [31].”
Training with Quantization Noise for Extreme Model Compression (2020, methodology): “Other authors have worked on lightweight architectures, by modifying existing models (Zhang et al., 2018; Wu et al., 2019; Sukhbaatar et al., 2019a) or developing new networks, such as MobileNet (Howard et al., 2019), ShufﬂeNet (Zhang et al., 2017), and EfﬁcientNet (Tan & Le, 2019) in vision.”
Meshed-Memory Transformer for Image Captioning (2019, background): “After the emergence of Convolutional language models, which have been explored for captioning as well [5], new fully-attentive paradigms [39, 8, 36] have been proposed and achieved state-of-the-art results in machine translation and language understanding tasks.”
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling (2021, background): “This similar relations have also been discussed in persistent memory [57] and non-parametric attention [35].”
Augmenting Transformers with KNN-Based Composite Memory for Dialog (2020, neutral): “For example, Memory Networks (We-ston et al., 2014; Sukhbaatar et al., 2015, 2019) introduce attention mechanisms over large external memories.”
RSTNet: Captioning with Adaptive Attention on Visual and Non-Visual Words (2021, background): “In order to address this problem, [37, 7, 33] proposed to replace recurrence and convolutions with attention mechanisms and excitedly refreshed almost all the metrics of neural language processing (NLP).”
Lite Transformer with Long-Short Range Attention (2020, background): “Recently, there have been a lot of variants to the transformer (Ahmed et al., 2017; Ott et al., 2018; Chen et al., 2018; Paulus et al., 2018; Shaw et al., 2018; Sukhbaatar et al., 2019a;b; Child et al., 2019).”
Adaptive Transformers in RL (2020, methodology): “b environment we used. We expect this to amplify the performance beneﬁts we found when using adaptive span in the Stable Transformers. Two additional future experiments are to use 5 persistent memory [8] in the Stable Transformer as well as GRU gating as was done in [1]. We include the code to easil”
HyperPrompt: Prompt-based Task-Conditioning of Transformers (2022, background): “Hyper-prompts are injected to the keys and values in the self-attention module, reminiscent of memory augmented Transformers (Sukhbaatar et al., 2019).”
Scale down Transformer by Grouping Features for a Lightweight Character-level Language Model (2020, background): “The ﬁrst is to restrict dependencies between input tokens to reduce superﬂuous pair-wise calculations (Child et al., 2019; Guo et al., 2019b; Sukhbaatar et al., 2019a).”
A Survey of Transformers (2021, methodology): “[127] demonstrate that replacing the ReLU activation with Softmax and dropping the bias term in FFN effectively turns FFN into an attention module where position-wise inputs attend to a global key-value memory of Dffn slots.”
Memorizing Transformers (2022, background): “Sukhbaatar et al. (2019) make the observation that the feed-forward portion of a transformer layer functions very much like attention if one replaces the ReLU activation with softmax.”