Memory-based Parameter Adaptation

By Pablo Sprechmann, Siddhant M. Jayakumar, Jack W. Rae, Alexander Pritzel, Adrià Puigdomènech Badia, Benigno Uria, Oriol Vinyals, Demis Hassabis, Razvan Pascanu, Charles Blundell. 2018

106 citations recorded by the source corpus.

Deep neural networks have excelled on a wide range of problems, from vision to language and game playing. Neural networks very gradually incorporate information into weights as they process data, requiring very low learning rates. If the training distribution shifts, the network is slow to adapt, and when it does adapt, it typically performs badly on the training distribution before the shift. Our method, Memory-based Parameter Adaptation, stores examples in memory and then uses a context-based lookup to directly modify the weights of a neural network. Much higher learning rates can be used for this local adaptation, reneging the need for many iterations over similar data before good predictions can be made. As our method is memory-based, it alleviates several shortcomings of neural networks, such as catastrophic forgetting, fast, stable acquisition of new knowledge, learning with an imbalanced class labels, and fast learning during evaluation. We demonstrate this on a range of supervised tasks: large-scale image classification and language modelling.

Subfields: Artificial Intelligence, Radiology, Nuclear Medicine and Imaging, Model Architectures, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Radiology, Nuclear Medicine and Imaging

In Model Architectures

In Meta & Continual Learning

Citations in the corpus

CrossTransformers: spatially-aware few-shot transfer (2020, background): “In the limit, attention over long-term experiences accumulated in memories [43, 56, 69, 76] can augment more traditional learning.”
Meta-learning with differentiable closed-form solvers (2018, methodology): “…typical of deep neural networks (McCloskey & Cohen, 1989), several recent methods (Santoro et al., 2016; Kaiser et al., 2017; Munkhdalai & Yu, 2017; Sprechmann et al., 2018) make use of memory-augmented models, which can ﬁrst retain and then access important and previously unseen information…”
Policy Consolidation for Continual Reinforcement Learning (2019, methodology): “…ones that allocate more neural resources over time (Ring, 1997; Rusu et al., 2016), and ones that explicitly store (or train generative models to mimic) past data which are used in various ways to improve the memory of the network (Isele & Cosgun, 2018; Sprechmann et al., 2018; Shin et al., 2017).”
Class Gradient Projection For Continual Learning (2022, methodology): “Some methods [27, 33, 38] use replayed samples from previous tasks to constrain the parameters’ update when learning the new task.”
Continual Learning in Task-Oriented Dialogue Systems (2020, background): “Continual Learning in NLP has been explore in both in classification (d’Autume et al., 2019; Sprechmann et al., 2018; Wang et al., 2020) and generation (Sun et al., 2019; Hu et al., 2020) tasks.”
Embracing Change: Continual Learning in Deep Neural Networks. (2020, methodology): “In this category are also episodic memory methods, which are distinguished from re-hearsal methods because they not only use replay memory for training, but also for inference [60 – 62].”
Efficient Meta Lifelong-Learning with Limited Memory (2020, methodology): “…to its potential practical applications, there is a surge of research interest in the lifelong learning, especially in the vision domain (Rusu et al., 2016; Kirkpatrick et al., 2017; Zenke et al., 2017; Lopez-Paz and Ranzato, 2017; Yoon et al., 2018; Sprechmann et al., 2018; Chaudhry et al., 2019).”
Continual Learning by Asymmetric Loss Approximation With Single-Side Overestimation (2019, neutral): “Existing methods [24, 34, 15, 35, 13] often keep track of the data in the previous tasks by allocating additional memory space called episodic memory or by generating training examples using generative adversarial networks [11], but they suffer from substantial memory demands to store the informatio”
Episodic Memory in Lifelong Language Learning (2019, methodology): “In local adaptation, we follow Memory-based Parameter Adaptation (MbPA; Sprechmann et al., 2018) and use examples retrieved from memory to update model parameters used to make a prediction of a particular test example.”
Task-Free Continual Learning (2018, methodology): “Then there are methods using an episodic memory, such as iCARL (incremental Classiﬁer and Representation Learning) [29] and Memory Based Parameter Adaptation [35].”
Incremental Few-Shot Learning with Attention Attractor Networks (2018, methodology): “Truncated BPTT is a popular learning algorithm in many recent meta-learning approaches [2, 27, 7, 34, 3].”
Continual Learning for Robotics (2019, background): “The use of dual architecture is explicit in many bio-inspired approaches such as [48, 51, 115, 154, 70].”
Dynamic Evaluation of Neural Sequence Models (2017, background): “For instance, Sprechmann et al. (2018) reported using dynamic evaluation with optimisation parameters obtained during training, and achieved minimal test time improvements.”
Fast Parametric Learning with Activation Memorization (2018, background): “, 2016b) and (b) Memory-based Parameter Adaptation (MbPA) (Sprechmann et al., 2018).”
Meta-Learning with Sparse Experience Replay for Lifelong Language Learning (2020, background): “, 2018) that prevent large updates on weights that are important to previously seen tasks; (2) memorybased approaches (Rebuffi et al., 2017; Sprechmann et al., 2018; Wang et al., 2019; d’Autume et al., 2019) that replay examples stored in the memory; (3) generative replay-based approaches (Shin et a”
A Survey on Negative Transfer (2020, background): “Existing literature mainly studies how to mitigate catastrophic forgetting using regularization [113], [114], memory replay [115]–[117], parameter isolation [118], [119], etc., whereas forward NT in lifelong learning is less investigated [120].”
O N M EMORY IN H UMAN AND A RTIFICIAL L ANGUAGE P ROCESSING S YSTEMS (2020, background): “Although previous work has explored separating computation and storage ( e.g. , Graves et al., 2014; Weston et al., 2015; Sprechmann et al., 2018), the dominant approach focuses on building large neural models where both computation and storage are modeled implicitly in model parameters (i.e.,…”
Memory, Show the Way: Memory Based Few Shot Word Representation Learning (2018, unknown): “…based Meta Learning Memory augmented neural networks (MANN) are widely used in different tasks for efﬁcient recall of experience and fast adaptation to new knowledge (Bahdanau et al., 2014; Merity et al., 2017; Miller et al., 2016; Grave et al., 2017; Sprechmann et al., 2018; Wang et al., 2017).”