Improving Neural Language Models with a Continuous Cache

By Edouard Grave, Armand Joulin, Nicolas Usunier. 2016

309 citations recorded by the source corpus.

We propose an extension to neural network language models to adapt their prediction to the recent history. Our model is a simplified version of memory augmented networks, which stores past hidden activations as memory and accesses them through a dot product with the current hidden activation. This mechanism is very efficient and scales to very large memory sizes. We also draw a link between the use of external memory in neural network and cache models used with count based language models. We demonstrate on several language model datasets that our approach performs significantly better than recent memory augmented networks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Character-Level Language Modeling with Deeper Self-Attention (2018, background): “A closely related architecture is the Neural Cache Model [12], where the RNN is allowed to attend to all of its previous hidden states at each step.”
Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “Later, the amount of tokens used to compute this distribution was extended to a much larger memory by leveraging approximate nearest neighbors search (Grave et al., 2017a).”
Learning and Evaluating General Linguistic Intelligence (2019, background): “Recent work on neural cache (Grave et al., 2017) and dynamic evaluation (Krause et al., 2018) provide some evidence of the effectiveness of this approach for language modeling.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, methodology): “We adopted adaptive softmax and input representations (Baevski & Auli, 2018; Grave et al., 2016a).”
Universal Transformers (2018, background): “Model LM Perplexity & (Accuracy) RC Accuracy control dev test control dev test Neural Cache [10] 129 139 Dhingra et al.”
Improving language models by retrieving from trillions of tokens (2021, background): “Continuous cache (Grave et al., 2017) adds probability mass to tokens for which previous activations resemble the current activation vector, extending the model’s context to the local history.”
Compressive Transformers for Long-Range Sequence Modelling (2019, background): “This dataset has been a popular recent LM benchmark due to the potential to exploit longer-range dependencies (Grave et al., 2016; Rae et al., 2018; Bai et al., 2018b).”
Efficient Content-Based Sparse Attention with Routing Transformers (2020, neutral): “LSTMs (Grave et al., 2017) 40.”
Augmented Language Models: a Survey (2023, background): “LMs can be augmented with memory units, for example via a neural cache of recent inputs (Grave et al., 2017; Merity et al., 2017), to improve their reasoning abilities.”
Generalization through Memorization: Nearest Neighbor Language Models (2019, methodology): “…across all its occurrences in the retrieved targets (items that do not appear in the retrieved targets have zero probability): Finally, we follow Grave et al. (2017a) and interpolate the nearest neighbor distribution p kNN with the model distribution p LM using a tuned parameter λ to produce…”
Large Memory Layers with Product Keys (2019, background): “Neural cache models [14] suffer from the same scaling issues.”
Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion (2022, background): “In the language modeling space, there is a large body of work on nearest neighbor and cache-based language modeling (Khandelwal et al., 2020; Grave et al., 2017; Merity et al., 2017; Khandelwal et al., 2021; Yogatama et al., 2021) for accessing a large set of documents.”
Retrieval Augmentation Reduces Hallucination in Conversation (2021, methodology): “…a retrievalaugmentation step includes the area of language modeling, where it is used for pre-training (Guu et al., 2020), and as a memory (Yogatama et al., 2021), especially using k-nearest neighbor-based cache models (Khandelwal et al., 2021, 2020; Grave et al., 2016; Merity et al., 2016).”
Adaptive Semiparametric Language Models (2021, background): “context to a neural network—such as those found in neural cache (Grave et al., 2017c), transformerXL (Dai et al.”
Unbounded cache model for online language modeling with open vocabulary (2017, methodology): “Our approach is inspired by the cache model of Khun [36] and can be seen as an extension of Grave et al. [22] to an unbounded memory structure tailored to deal with out-of-vocabulary and rare words.”
Language Modeling with Gated Convolutional Networks (2016, methodology): “…et al., 2016) 30.6 32 GPUs BIG GLSTM-G4 (Kuchaiev & Ginsburg, 2017) 23.3 ∗ 8 GPUs LSTM-2048 (Grave et al., 2016a) 43.9 1 GPU 2-layer LSTM-2048 (Grave et al., 2016a) 39.8 1 GPU GCNN-13 38.1 1 GPU GCNN-14 Bottleneck 31.9 8 GPUs (Jozefowicz et al., 2016) which uses the full softmax, the…”