Continual Reinforcement Learning with Complex Synapses

By Christos Kaplanis, Murray Shanahan, Claudia Clopath. 2018

102 citations recorded by the source corpus.

Unlike humans, who are capable of continual learning over their lifetimes, artificial neural networks have long been known to suffer from a phenomenon known as catastrophic forgetting, whereby new learning can lead to abrupt erasure of previously acquired knowledge. Whereas in a neural network the parameters are typically modelled as scalar values, an individual synapse in the brain comprises a complex network of interacting biochemical components that evolve at different timescales. In this paper, we show that by equipping tabular and deep reinforcement learning agents with a synaptic model that incorporates this biological complexity (Benna & Fusi, 2016), catastrophic forgetting can be mitigated at multiple timescales. In particular, we find that as well as enabling continual learning across sequential training of two simple tasks, it can also be used to overcome within-task forgetting by reducing the need for an experience replay database.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Experience Replay for Continual Learning (2018, background): “There has recently been a surge of interest in methods for preventing catastrophic forgetting in RL, inspired in part by neuroscience [11, 13, 27, 29].”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “Similarly, resolving within-task catastrophic forgetting can facilitate reinforcement learning [299], [300] and stabilize the training of GANs [301], [302].”
Synaptic weight decay with selective consolidation enables fast learning without catastrophic forgetting (2019, methodology): “The synaptic cascade model (Fusi et al., 2005; Benna and Fusi, 2016; Kaplanis et al., 2018) postulates hidden states that, different from our model, do not contribute to the effective synaptic strength.”
Disentangling Transfer in Continual Reinforcement Learning (2022, background): “The authors of [20] show how the synaptic Benna-Fusi model can be added on top of value-based RL methods to mitigate forgetting at both intra-and inter-task scales.”
Continuous Coordination As a Realistic Scenario for Lifelong Learning (2021, background): “RL can be a natural ﬁt for studying LLL as it provides an agent-environment interaction paradigm wherein the agent is exposed to non-stationary streams of data (Kaplanis et al., 2018; 2019).”
Online Continual Learning in Image Classification: An Empirical Survey (2021, background): “Although CL is also studied in reinforcement learning [19, 20, 21] and more recently in unsupervised learning [22], the image classification problem is still the main focus for many CL researchers.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Additionally, it may not be necessary to perform replay to combat time correlation during learning (Kaplanis, Shanahan, and Clopath, 2018).”
Embracing Change: Continual Learning in Deep Neural Networks. (2020, background): “Some continual learning approaches have successfully demonstrated simpli ﬁ ed mechanisms built on such principles [4,5], but many core capabilities remain elusive.”
Continual Reinforcement Learning with Multi-Timescale Replay (2020, background): “Other non-replay-based approaches that use multiple timescales of memory to mitigate catastrophic forgetting in deep RL include [Kaplanis et al., 2018], which consolidates the individual parameters at multiple timescales, and [Kaplanis et al., 2019], which consolidates the agent’s policy at…”
Continual Learning with Adaptive Weights (CLAW) (2019, background): “(b) The architecture-based approach aims to deal with stability and adaptation issues by a fixed division of the architecture into global and local parts (Rusu et al., 2016b; Fernando et al., 2017; Shin et al., 2017; Kaplanis et al., 2018; Xu & Zhu, 2018; Yoon et al., 2018; Li et al., 2019b).”
Orthogonal Gradient Descent for Continual Learning (2019, methodology): “These approaches are well motivated by neuro-biological models of memory Fusi et al. (2005); Kaplanis et al. (2018) and are computationally fast and do not require storing data.”
Continual Reinforcement Learning in 3D Non-stationary Environments (2019, methodology): “Secondly by integrating more accurate synaptic plasticity models as shown in [19, 5] and going beyond mere consolidation processes which tend to quickly saturate the model learning capacity.”
Functional Regularisation for Continual Learning using Gaussian Processes (2019, background): “While knowledge of task boundaries is assumed given in most aforementioned methods, the work in (Kaplanis et al., 2018) circumvents this assumption by taking inspiration from alternative synaptic models, essentially mitigating catastrophic forgetting on multiple timescales.”
Continual Representation Learning for Biometric Identification (2020, background): “Existing continual learning works focus on general object recognition [7, 29, 48], object detection [16, 54]), image generation [33, 68], reinforcement learning tasks [1, 27, 72] and unsupervised learning tasks [12].”
Artificial Neural Networks for Neuroscientists: A Primer (2020, background): “Models of such complex synapses are differentiable and therefore can be incorporated into ANNs (Kaplanis et al., 2018).”
Meta-learning of Sequential Strategies (2019, methodology): “A relaxation of this approach, e.g. explored by Kaplanis et al. [2018], would be to measure how fast one recovers performance, which converts continual learning into a meta-learning problem.”
Biological underpinnings for lifelong learning machines (2022, background): “67 has also been shown to mitigate forgetting in a reinforcement learning paradigm 182 .”
Neural networks with late-phase weights (2020, background): “, 2017), and continual learning with biologically-realistic synapses (Kaplanis et al., 2018; Leimer et al., 2019), to name a few.”