Learning to Remember Rare Events

By Lukasz Kaiser, Ofir Nachum, Aurko Roy, Samy Bengio. 2017

388 citations recorded by the source corpus.

Despite recent advances, memory-augmented deep neural networks are still limited when it comes to life-long and one-shot learning, especially in remembering rare events. We present a large-scale life-long memory module for use in deep learning. The module exploits fast nearest-neighbor algorithms for efficiency and thus scales to large memory sizes. Except for the nearest-neighbor query, the module is fully differentiable and trained end-to-end with no extra supervision. It operates in a life-long manner, i.e., without the need to reset it during training. Our memory module can be easily added to any part of a supervised neural network. To show its versatility we add it to a number of networks, from simple convolutional ones tested on image classification to deep sequence-to-sequence and recurrent-convolutional models. In all cases, the enhanced network gains the ability to remember and do life-long one-shot learning. Our module remembers training examples shown many thousands of steps in the past and it can successfully generalize from them. We set new state-of-the-art for one-shot learning on the Omniglot dataset and demonstrate, for the first time, life-long one-shot learning in recurrent neural networks on a large-scale machine translation task.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks (2017, neutral): “(Kaiser et al., 2017) 98 .”
Generalization through Memorization: Nearest Neighbor Language Models (2019, background): “More recently, Kaiser et al. (2017) have used a similarly differentiable memory that is learned and updated during training, and is applied to one-shot learning tasks.”
Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing (2021, neutral): “There are a wide variety of methods to achieve few-shot learning including model agnostic meta-learning [29] (learning features rapidly adaptable to new tasks), embedding learning [8] (embedding each sample in a lower-dimensional space where similar samples are close together), memorybased learning ”
CrossTransformers: spatially-aware few-shot transfer (2020, background): “In the limit, attention over long-term experiences accumulated in memories [43, 56, 69, 76] can augment more traditional learning.”
Meta-learning with differentiable closed-form solvers (2018, methodology): “…the drawback of catastrophic forgetting typical of deep neural networks (McCloskey & Cohen, 1989), several recent methods (Santoro et al., 2016; Kaiser et al., 2017; Munkhdalai & Yu, 2017; Sprechmann et al., 2018) make use of memory-augmented models, which can ﬁrst retain and then access…”
Task Agnostic Meta-Learning for Few-Shot Learning (2018, neutral): “[21] 98.4% 99.6% Prototypical Nets [19] 98.8% 99.7% Meta Nets [16] 98.9% - Snail [14] 99.07 ± 0.16% 99.78 ± 0.09% MAML [2] 98.”
Meta Networks (2017, neutral): “Pixel kNN (Kaiser et al., 2017) 41.”
Learning to Compare: Relation Network for Few-Shot Learning (2017, methodology): “RNN Memory Based Another category of approaches leverage recurrent neural networks with memories [26, 31].”
Few-Shot Learning with Graph Neural Networks (2017, neutral): “Pair-Wise Mehrotra & Dukkipati (2017) - - 94.8% - Prototypical Networks Snell et al. (2017) 97.4% 99.3% 95.4% 98.8% ConvNet with Memory Kaiser et al. (2017) 98.4% 99.6% 95.0% 98.6% Agnostic Meta-learner Finn et al. (2017) 98.7 ± 0.4% 99.9 ± 0.3% 95.8 ± 0.3% 98.9 ± 0.2% Meta Networks Munkhdalai & Yu…”
Deep Meta-Learning: Learning to Learn in the Concept Space (2018, background): “(Kaiser et al., 2017) also learn a matching network like (Vinyals et al., 2016) but include a memory module that retains previously seen examples or their representatives.”
Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “However, some attempts have been made (e.g. Peris and Casacuberta, 2018; Liu et al., 2018; Kaiser et al., 2017; Tu et al., 2018).”
Scalable Recollections for Continual Lifelong Learning (2017, methodology): “In our work we find that selecting subsets works well although a buffer may be even more efficient with online clustering strategies as in (Kaiser et al. 2017).”
Memory-based Parameter Adaptation (2018, methodology): “Variants of this architecture have been successfully used in the context of classi-ﬁcation (Vinyals et al., 2016; Santoro et al., 2016; Kaiser et al., 2017), language modelling (Merity et al., 2016; Grave et al., 2016), reinforcement learning (Blundell et al., 2016; Pritzel et al., 2017), machine…”
Abstractive Summarization of Reddit Posts with Multi-level Memory Networks (2018, background): “Many memory network models have been proposed to improve memorization capability of neural networks (Kaiser et al., 2017; Na et al., 2017; Yoo et al., 2019).”
Few-Shot Learning Through an Information Retrieval Lens (2017, methodology): “Further, [19] has a metric-learning flavor but employs external memory as a vehicle for remembering representations of rarely-observed classes.”
Mem2Seq: Effectively Incorporating Knowledge Bases into End-to-End Task-Oriented Dialog Systems (2018, methodology): “Similarly, memory encoders have been used in neural machine translation (Wang et al., 2016), and meta-learning application (Kaiser et al., 2017).”
Boosting Few-Shot Visual Learning With Self-Supervision (2019, neutral): “, training images, of a few-shot task [26, 33, 53, 55, 57, 59]; methods learning to map a test example to a class label by accessing memory modules that store training examples for that task [11, 25, 35, 38, 50]; approaches that learn how to generate the weights of a classifier [12, 16, 43, 44, 13] ”