Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity

By Thomas Miconi, Aditya Rawal, Jeff Clune, Kenneth O. Stanley. 2018

106 citations recorded by the source corpus.

The impressive lifelong learning in animal brains is primarily enabled by plastic changes in synaptic connectivity. Importantly, these changes are not passive, but are actively controlled by neuromodulation, which is itself under the control of the brain. The resulting self-modifying abilities of the brain play an important role in learning and adaptation, and are a major basis for biological reinforcement learning. Here we show for the first time that artificial neural networks with such neuromodulated plasticity can be trained with gradient descent. Extending previous work on differentiable Hebbian plasticity, we propose a differentiable formulation for the neuromodulation of plasticity. We show that neuromodulated plasticity improves the performance of neural networks on both reinforcement learning and supervised learning tasks. In one task, neuromodulated plastic LSTMs with millions of parameters outperform standard LSTMs on a benchmark language modeling task (controlling for the number of parameters). We conclude that differentiable neuromodulation of plasticity offers a powerful new framework for training neural networks.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Learning to Continually Learn (2020, methodology): “Another is combining ANML with recent work into meta-learning with differentiable Heb-bian learning [33], including differentiable neuromodulated Hebbian plasticity [34].”
Meta-Learning with Warped Gradient Descent (2019, methodology): “(Miconi et al., 2019).”
AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence (2019, background): “As with Hebbian learning, there is a long history of neuromodulation work without using gradient descent, but we have recently introduced the ability to train networks with neuromodulation in a fully differentiable, end-to-end fashion [113].”
Meta-learnt priors slow down catastrophic forgetting in neural networks (2019, background): “While the learning rule was not learnt, separate works have explored such possibility, for example in Backpropamine [21], where gradient descent is performed on a differentiable Hebbian learning rule in a meta-learning context.”
Linear Transformers Are Secretly Fast Weight Programmers (2021, background): “…Schmidhuber, 2017), also under different names, e.g., hypernetworks (Ha et al., 2017; Perez et al., 2018; Galanti & Wolf, 2020), dynamic plasticity (Miconi et al., 2018; 2019), dynamic convolution (Klein et al., 2015; Noh et al., 2016; Jia et al., 2016), or lambda networks (Bello, 2021) used for…”
Discovering Reinforcement Learning Algorithms (2020, methodology): “Other approaches include simple approximation [25], RNNs with Hebbian learning [21, 22], and gradient preconditioning [12].”