Differentiable plasticity: training plastic neural networks with backpropagation

By Thomas Miconi, Jeff Clune, Kenneth O. Stanley. 2018

196 citations recorded by the source corpus.

How can we build agents that keep learning from experience, quickly and efficiently, after their initial training? Here we take inspiration from the main mechanism of learning in biological brains: synaptic plasticity, carefully tuned by evolution to produce efficient lifelong learning. We show that plasticity, just like connection weights, can be optimized by gradient descent in large (millions of parameters) recurrent networks with Hebbian plastic connections. First, recurrent plastic networks with more than two million parameters can be trained to memorize and reconstruct sets of novel, high-dimensional 1000+ pixels natural images not seen during training. Crucially, traditional non-plastic recurrent networks fail to solve this task. Furthermore, trained plastic networks can also solve generic meta-learning tasks such as the Omniglot task, with competitive results and little parameter overhead. Finally, in reinforcement learning settings, plastic networks outperform a non-plastic equivalent in a maze exploration task. We conclude that differentiable plasticity may provide a powerful novel approach to the learning-to-learn problem.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Electrical and Electronic Engineering, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Electrical and Electronic Engineering

In Optimization & Training

Citations in the corpus

Learning to Continually Learn (2020, methodology): “Another is combining ANML with recent work into meta-learning with differentiable Heb-bian learning [33], including differentiable neuromodulated Hebbian plasticity [34].”
Episodic Memory in Lifelong Language Learning (2019, background): “3 Future work can explore cheaper alternatives to gradient-based updates for local adaptation (e.g., a Hebbian update similar to the update that is used in plastic networks; Miconi et al., 2018).”
Meta-Learning in Neural Networks: A Survey (2020, background): “Meta-learning can define the function class of v so as to lead to the discovery of novel learning rules that are unsupervised [16] or biologically plausible [46], [278], [279], making use of ideas less commonly used in contemporary deep learning such as Hebbian updates [278] and neuromodulation [279”
Stabilizing Transformers for Reinforcement Learning (2019, background): “Others have looked at improving performance of LSTM agents by extending the architecture with stacked hierarchical connections / multiple temporal scales and auxiliary losses (Jaderberg et al., 2019; Stooke et al., 2019) or allowing an inner-loop update to the RNN weights (Miconi et al., 2018).”
Random Feature Attention (2021, background): “There are interesting connections between gated R FA and fast weights (Schmidhuber, 1992; 1993; Ba et al., 2016; Miconi et al., 2018, inter alia ).”
Meta-Learning with Warped Gradient Descent (2019, methodology): “To illustrate both how WarpGrad may be used with Recurrent Neural Networks in an online metalearning setting, as well as in a Reinforcement Learning environment, we evaluate it in a maze navigation task proposed by [29].”
Meta-Learning with Sparse Experience Replay for Lifelong Language Learning (2020, methodology): “Using Taylor series approximation as in Nichol et al. (2018), the expected gradient under minibatch sampling could be expressed as:”
Artificial Neural Networks for Neuroscientists: A Primer (2020, background): “In the first kind, the effective connection matrix W =W +A is the sum of two connection matrices, W trained by SGD, and A driven by Hebbian plasticity (Ba et al., 2016a; Miconi et al., 2018),”
Linear Transformers Are Secretly Fast Weight Programmers (2021, background): “…Schmidhuber, 2017), also under different names, e.g., hypernetworks (Ha et al., 2017; Perez et al., 2018; Galanti & Wolf, 2020), dynamic plasticity (Miconi et al., 2018; 2019), dynamic convolution (Klein et al., 2015; Noh et al., 2016; Jia et al., 2016), or lambda networks (Bello, 2021) used for…”
Neuroevolution of self-interpretable agents (2020, background): “ing [44] rule inspired by neurons that fire together wire together, shown to be useful for associative learning [4, 63].”
ProMP: Proximal Meta-Policy Search (2018, methodology): “Another set of methods embeds the structure of a classical learning algorithm in the meta-learning procedure, and optimizes the parameters of the embedded learner during the meta-training (Hüsken & Goerick, 2000; Finn et al., 2017; Nichol et al., 2018; Miconi et al., 2018).”
Learning to Reason with Third-Order Tensor Products (2018, background): “Variations of such outer product-based Fast Weights were able to generalise in a variety of small but complex sequence problems where standard RNNs tend to perform poorly [27, 31, 40].”
Introducing Symmetries to Black Box Meta Reinforcement Learning (2021, background): “…In the supervised and reinforcement learning contexts, learned learning rules (Bengio et al. 1992) or fast weights (Schmidhu-ber 1992, 1993a; Miconi, Stanley, and Clune 2018; Schlag, Munkhdalai, and Schmidhuber 2021; Najarro and Risi 2020) describe (meta-)learned mechanisms (slow weights)…”
Hindsight Foresight Relabeling for Meta-Reinforcement Learning (2021, background): “Prior works have attempted to represent the RL process using a recurrent neural network (RNN) (Duan et al., 2016; Miconi et al., 2018; Wang et al., 2016) - the hidden state is maintained across episode boundaries and informs the policy as to what task it is currently solving.”
Discovering Reinforcement Learning Algorithms (2020, methodology): “Other approaches include simple approximation [25], RNNs with Hebbian learning [21, 22], and gradient preconditioning [12].”
Transferring Knowledge across Learning Processes (2018, background): “These regularizers operate by protecting the parameters that affect the loss function the most (Miconi et al., 2018; Zenke et al., 2017; Kirkpatrick et al., 2017; Lee et al., 2017; Serrà et al., 2018).”
Long short-term memory and Learning-to-learn in networks of spiking neurons (2018, background): “[12] Thomas Miconi, Jeff Clune, and Kenneth O.”