Frustratingly Short Attention Spans in Neural Language Modeling

By Michal Daniluk, Tim Rocktaschel, Johannes Welbl, Sebastian Riedel. 2017

119 citations recorded by the source corpus.

Neural language models predict the next token using a latent representation of the immediate token history. Recently, various methods for augmenting neural language models with an attention mechanism over a differentiable memory have been proposed. For predicting the next token, these models query information from a memory of the recent history which can facilitate learning mid- and long-range dependencies. However, conventional attention mechanisms used in memory-augmented neural language models produce a single output vector per time step. This vector is used both for predicting the next token as well as for the key and value of a differentiable memory of a token history. In this paper, we propose a neural language model with a key-value attention mechanism that outputs separate representations for the key and value of a differentiable memory, as well as for encoding the next-word distribution. This model outperforms existing memory-augmented neural language models on two corpora. Yet, we found that our method mainly utilizes a memory of the five most recent output representations. This led to the unexpected main finding that a much simpler model based only on the concatenation of recent output representations from previous time steps is on par with more sophisticated memory-augmented neural language models.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Character-Level Language Modeling with Deeper Self-Attention (2018, methodology): “Another similar model is used in [9] where a key-value attention mechanism similar to transformers is used.”
Long Range Arena: A Benchmark for Efficient Transformers (2020, background): “LSTM language models augmented with attention have been shown to rarely attend beyond seven preceding words of context (Daniluk et al., 2017) and samples from LSTM language models are known to quickly devolve into generic text.”
Lightweight and Efficient Neural Natural Language Processing with Quaternion Networks (2019, background): “Notably, the idea of splitting vectors has also been explored (Daniluk et al., 2017), which is in similar spirit to breaking a vector into four components.”
Do Transformers Need Deep Long-Range Memory? (2020, background): “For regular language modelling, Daniluk et al. (2017) observed that an LSTM augmented with attention would rarely attend beyond seven preceding words of context.”
Linguistic Knowledge as Memory for Recurrent Neural Networks (2017, unknown): “Miller et al. (2016) and Daniluk et al. (2017) argue that overloaded use of state representations as both memory content and address makes training of the network difﬁ-cult, and decompose these two functions by parameterizing them separately.”
Neural Models for Reasoning over Multiple Mentions Using Coreference (2018, background): “Attention mechanisms alleviate part of the issue, but empirical studies suggest RNNs with attention also have difficulty modeling long-term dependencies (Daniluk et al., 2017).”
Memory Architectures in Recurrent Neural Network Language Models (2018, methodology): “Our results generally agree with Daniluk et al. (2017) that show that increasing the attention size generally improves performance up to a certain threshold.”
A General Survey on Attention Mechanisms in Deep Learning (2022, unknown): “The notation introduced here is based on the notation that was introduced in [23] and popularized in [13].”
Lost in the Middle: How Language Models Use Long Contexts (2023, background): “In a similar vein, Daniluk et al. (2017) find that attentive LSTM language models tend to mainly use recent history.”
O N M EMORY IN H UMAN AND A RTIFICIAL L ANGUAGE P ROCESSING S YSTEMS (2020, background): “Separating neural network components based on their functions has been found useful in other contexts, e.g. , for attention with a query, key, and value component (Daniluk et al., 2017).”
Artificial Intelligence for the Metaverse: A Survey (2022, background): “DL has been further exploited to overcome the learning limitation of conventional ML algorithms and effectively deal with many challenging tasks in NLP.”
Hopfield Networks is All You Need (2020, background): “EMN with dot products became very popular and implement a key-value attention [21] for self-attention.”
EEG-Based Emotion Recognition via Channel-Wise Attention and Self Attention (2023, background): “In addition, the self-attention mechanism is popular because it can improve long-range dependency modeling [42].”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, methodology): “[56] tested several networks using variations of attention mechanisms.”
Bidirectional LSTM with attention mechanism and convolutional layer for text classification (2019, background): “Daniluk et al. [48] proposed a neural language model with a key-value attention mechanism that outputs separate representations for the key and value of a differentiable memory, as well as for encoding the next-word distribution.”
Attention in Natural Language Processing (2019, background): “[61], who use different representations of the input for computing the attention distribution and the contextual information.”