Unbounded cache model for online language modeling with open vocabulary

By Edouard Grave, Moustapha Cissé, Armand Joulin. 2017

78 citations recorded by the source corpus.

Recently, continuous cache models were proposed as extensions to recurrent neural network language models, to adapt their predictions to local changes in the data distribution. These models only capture the local context, of up to a few thousands tokens. In this paper, we propose an extension of continuous cache models, which can scale to larger contexts. In particular, we use a large scale non-parametric memory component that stores all the hidden activations seen in the past. We leverage recent advances in approximate nearest neighbor search and quantization algorithms to store millions of representations while searching them efficiently. We conduct extensive experiments showing that our approach significantly improves the perplexity of pre-trained language models on new distributions, and can scale efficiently to much larger contexts than previously proposed local cache models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “Later, the amount of tokens used to compute this distribution was extended to a much larger memory by leveraging approximate nearest neighbors search (Grave et al., 2017a).”
Generalization through Memorization: Nearest Neighbor Language Models (2019, methodology): “…across all its occurrences in the retrieved targets (items that do not appear in the retrieved targets have zero probability): Finally, we follow Grave et al. (2017a) and interpolate the nearest neighbor distribution p kNN with the model distribution p LM using a tuned parameter λ to produce…”
Large Memory Layers with Product Keys (2019, methodology): “Neural cache models [15] suffer from the same scaling issues, which are circumvented by adopting approximate lookup techniques at test time [14].”
Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context (2018, background): “However, the success of copy mechanisms such as attention and caching (Bahdanau et al., 2015; Hill et al., 2016; Merity et al., 2017; Grave et al., 2017a,b) suggests that information in the distant context is very useful.”
Fast Parametric Learning with Activation Memorization (2018, methodology): “Although the cache is of fixed size n, it can be defined to be very large with sparse attention and efficient data-structures (Rae et al., 2016; Kaiser et al., 2017; Grave et al., 2017).”
Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling (2021, methodology): “The cache model has been adopted for improving language generation in kNN-LMs [32], Unbounded Cache [20] and Pointer Sentinel Mixture Models [46].”
Training Language Models with Memory Augmentation (2022, methodology): “Our objective is also closely connected to the objective used in Grave et al. (2017b); Khandelwal et al. (2020), which linearly interpolates the distribution of standard language modeling, and a distribution de-ﬁned by cache/external datastore, e.g., P ( w ∣ c ) = ( 1 − λ ) P lm ( w ∣ c )+ λP kNN (…”
Learning and the Unknown: Surveying Steps toward Open World Recognition (2019, background): “…Navarro et al. 2018), • Lifelong Learning (Chen and Liu 2016; Rebufﬁ et al. 2017), • Natural Language (Prakhya, Venkataram, and Kalita 2017; Doan and Kalita 2017; Grave, Cisse, and Joulin 2017), • Novelty Detection (Bodesheim et al. 2015; Lazzaretti et al. 2016; Schultheiss et al. 2017), •…”
A Simple Cache Model for Image Recognition (2018, methodology): “Relatedly, efficient nearest neighbor methods can be utilized to scale our model to effectively unbounded cache sizes which would be useful under online learning and/or evaluation scenarios [4].”
Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification (2022, methodology): “Different from existing methods, ours does not require extra training, but designs the adapter as a query-key cache model [30,45,18] from the few-shot dataset.”
Explaining and Improving Model Behavior with k Nearest Neighbor Representations (2020, background): “The neural cache (Grave, Joulin, and Usunier 2017) and re-lated unbounded neural cache (Grave, Cisse, and Joulin 2017) retrieve similar instances from the sequence history to better adapt language models to the context.”
Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale (2018, background): “We observe other organizations already developing broad-purpose models for domains such as language modeling [13, 14], object recognition [20], and object detection [39].”
Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey (2024, methodology): “Different from CLIP-Adapter, the weights of adapter is obtained in a training-free manner from a query-key cache model [140], [57] constructed from fewshot supervisions in a non-parametric manner.”