Neural Modularity Helps Organisms Evolve to Learn New Skills without Forgetting Old Skills

By Kai Olav Ellefsen, Jean-Baptiste Mouret, Jeff Clune. 2015

177 citations recorded by the source corpus.

A long-standing goal in artificial intelligence is creating agents that can learn a variety of different skills for different problems. In the artificial intelligence subfield of neural networks, a barrier to that goal is that when agents learn a new skill they typically do so by losing previously acquired skills, a problem called catastrophic forgetting. That occurs because, to learn the new task, neural learning algorithms change connections that encode previously acquired skills. How networks are organized critically affects their learning dynamics. In this paper, we test whether catastrophic forgetting can be reduced by evolving modular neural networks. Modularity intuitively should reduce learning interference between tasks by separating functionality into physically distinct modules in which learning can be selectively turned on or off. Modularity can further improve learning by having a reinforcement learning module separate from sensory processing modules, allowing learning to happen only in response to a positive or negative reward. In this paper, learning takes place via neuromodulation, which allows agents to selectively change the rate of learning for each neural connection based on environmental stimuli (e.g. to alter learning in specific locations based on the task at hand). To produce modularity, we evolve neural networks with a cost for neural connections. We show that this connection cost technique causes modularity, confirming a previous result, and that such sparsely connected, modular networks have higher overall performance because they learn new skills faster while retaining old skills more and because they have a separate reinforcement learning module. Our results suggest (1) that encouraging modularity in neural networks may help us overcome the long-standing barrier of networks that cannot learn new skills without forgetting old ones, and (2) that one benefit of the modularity ubiquitous in the brains of natural animals might be to alleviate the problem of catastrophic forgetting.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Ecology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Ecology

Citations in the corpus

Born to Learn: the Inspiration, Progress, and Future of Evolved Plastic Artificial Neural Networks (2017, background): “Encouraging modularity (Ellefsen et al., 2015; Durr et al., 2010) or augmenting evolving networks with a dedicated external memory component (Lüders et al.”
Learning to Continually Learn (2020, background): “There is prior work in machine learning that harnesses neuromodulation techniques [7, 16, 39, 40, 42], but in such work neuromodulation directly modulates learning rates, instead of the approach taken in this work of directly modulating activations and thus indirectly controlling learning.”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, background): “It likely would also have no implicit memory of this frontier due to the problem of catastrophic forgetting [17–20].”
Diffusion-based neuromodulation can eliminate catastrophic forgetting in simple neural networks (2017, methodology): “[16], and produced a few individuals that completely avoided catastrophic forgetting, but were not able to reliably produce perfect solutions across all runs.”
Meta-learnt priors slow down catastrophic forgetting in neural networks (2019, background): “Close to the scope of this paper is the work of Ellefsen et al. (2015) (Ellefsen et al., 2015), that proposes to optimize the connections of a neural network by evolution, with an added cost per connection.”
Continual and One-Shot Learning Through Neural Networks with Dynamic External Memory (2017, methodology): “The main test for the agents in this paper is the season task [11], which tests an agent’s ability to withstand catastrophic forgetting.”
Continual Learning through Evolvable Neural Turing Machines (2016, background): “In order to enable these evolving ANNs to learn during their lifetime, efforts have been made to allow them to adapt online and learn from past experience [10, 11, 12, 13, 14].”
Evolving Neural Turing Machines for Reward-based Learning (2016, background): “In order to enable these evolving ANNs to learn during their lifetime, efforts have been made to allow them to adapt online and learn from past experience [14, 4, 1, 2, 10].”
Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity (2018, background): “By activating plasticity only in the subset of neural weights relevant for the task currently being performed, knowledge stored in other weights about different tasks is left untouched, alleviating catastrophic forgetting (Ellefsen et al., 2015; Velez & Clune, 2017).”
AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence (2019, methodology): “Additionally, there is a recent increase in meta-learning the learning algorithms themselves [37, 191, 40, 187, 161] (discussed more in Section 2.2).”
Evolvability Search: Directly Selecting for Evolvability in order to Study and Produce It (2016, unknown): “Such properties in turn improve evolvability [5, 9, 11, 22].”
Modular Brain Networks (2015, background): “In a similar vein, Ellefsen et al. (2015) demonstrated that modular brain networks can help prevent catastrophic forgetting, that is, the loss of a previously learned skill upon learning a new one.”