Can Active Memory Replace Attention?

By Lukasz Kaiser, Samy Bengio. 2016

61 citations recorded by the source corpus.

Several mechanisms to focus attention of a neural network on selected parts of its input or memory have been used successfully in deep learning models in recent years. Attention has improved image classification, image captioning, speech recognition, generative models, and learning algorithmic tasks, but it had probably the largest impact on neural machine translation. Recently, similar improvements have been obtained using alternative mechanisms that do not focus on a single part of a memory but operate on all of it in parallel, in a uniform way. Such mechanism, which we call active memory, improved over attention in algorithmic tasks, image processing, and in generative modelling. So far, however, active memory has not improved over attention for most natural language processing tasks, in particular for machine translation. We analyze this shortcoming in this paper and propose an extended model of active memory that matches existing attention models on neural machine translation and generalizes better to longer sentences. We investigate this model and explain why previous active memory models did not succeed. Finally, we discuss when active memory brings most benefits and where attention can be a better choice.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Attention is All you Need (2017, methodology): “We trained our models on one machine with 8 NVIDIA P100 GPUs.”
Neural Machine Translation in Linear Time (2016, methodology): “…(Luong et al., 2015; Chung et al., 2016a); the Grid LSTM translation model (Kalchbrenner et al., 2016a) that uses a multi-dimensional architecture; the Extended Neural GPU model (Kaiser & Bengio, 2016) that has a convolutional RNN architecture; the ByteNet and the two Recurrent ByteNet variants.”
Tensor2Tensor for Neural Machine Translation (2018, methodology): “The Extended Neural GPU model (Kaiser and Bengio, 2016) used a recurrent stack of gated convolutional layers, while the ByteNet model (Kalchbrenner et al., 2016) did away with recursion and used left-padded convolutions in the decoder.”
Natural Language Inference over Interaction Space (2017, methodology): “The goal of reducing sequential computation lays the foundation of several recent work such as Extended Neural GPU(Kaiser & Bengio, 2016), ByteNet(Kalchbrenner et al., 2016), ConvS2S(Gehring et al., 2017), Transformer(Vaswani et al., 2017).”
One Model To Learn Them All (2017, methodology): “The results we achieve are similar to the ones task-speciﬁc models get without heavy tuning, e.g., on English-French translation we improve on the Extended Neural GPU results reported last year [16].”
Learning to Remember Rare Events (2017, methodology): “To test versatility of our memory module, we also add it to the Extended Neural GPU, a convolutional-recurrent model introduced by Kaiser & Bengio (2016).”
Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement (2018, background): “This desirable property of exact and parallel decoding however comes at the expense of potential performance degradation (Kaiser and Bengio, 2016).”
Depthwise Separable Convolutions for Neural Machine Translation (2017, methodology): “The model in [13] (Extended Neural GPU) used a recurrent stack of gated convolutional layers, while the model in [11] (ByteNet) did away with recursion and used left-padded convolutions in the decoder.”
Weighted Transformer Network for Machine Translation (2017, methodology): “Several architectures have been proposed to reduce the computational load associated with recurrence-based computation (Gehring et al., 2016; 2017; Kaiser & Bengio, 2016; Kalchbrenner et al., 2016).”
Fast Decoding in Sequence Models using Discrete Latent Variables (2018, methodology): “Fully convolutional neural machine translation without this bottleneck was ﬁrst achieved in (Kaiser & Bengio, 2016) and (Kalchbrenner et al., 2016).”
Discrete Autoencoders for Sequence Models (2018, methodology): “Actually, for reasons analogous to those in (Sutskever et al., 2014), we ﬁrst reverse the sequence c ( s ) , then add a separator symbol (#), and only then concatenate it with s , as depicted in Figure 2.”
Deep Reinforcement Learning: An Overview (2017, background): “…target, which, rather than copying the weights directly as in DQN, updates the soft target network weights θ ′ slowly to track the learned networks weights θ : The authors adapted batch normalization to handle the issue that the different components of the observation with different physical units.”
Neural Machine Translation: A Review (2019, background): “Kaiser et al. (2018) reduced the amount of sequential computation by learning a sequence of latent discrete variables which is shorter than the actual target sentence, and generating the final sentence from this latent representation in parallel.”
You Look Twice: GaterNet for Dynamic Filter Selection in CNNs (2018, unknown): “Two vectors are then computed from g′ : gα = σ ′(g′ ) and gβ = 1(g ′ > 0) where σ′ is the saturating sigmoid function [19, 16]: σ′(x) = max(0,min(1, 1.”