Evolving neuromodulatory topologies for reinforcement learning-like problems

By Andrea Soltoggio, Peter Dürr, Claudio Mattiussi, Dario Floreano. 2007

83 citations recorded by the source corpus.

Environments with varying reward contingencies constitute a challenge to many living creatures. In such conditions, animals capable of adaptation and learning derive an advantage. Recent studies suggest that neuromodulatory dynamics are a key factor in regulating learning and adaptivity when reward conditions are subject to variability. In biological neural networks, specific circuits generate modulatory signals, particularly in situations that involve learning cues such as a reward or novel stimuli. Modulatory signals are then broadcast and applied onto target synapses to activate or regulate synaptic plasticity. Artificial neural models that include modulatory dynamics could prove their potential in uncertain environments when online learning is required. However, a topology that synthesises and delivers modulatory signals to target synapses must be devised. So far, only handcrafted architectures of such kind have been attempted. Here we show that modulatory topologies can be designed autonomously by artificial evolution and achieve superior learning capabilities than traditional fixed-weight or Hebbian networks. In our experiments, we show that simulated bees autonomously evolved a modulatory network to maximise the reward in a reinforcement learning-like environment.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Born to Learn: the Inspiration, Progress, and Future of Evolved Plastic Artificial Neural Networks (2017, background): “One study (Soltoggio et al., 2007) suggested that evolution discovers learning by discrete stepping stones: when evolution casually discovers a weak mechanism of learning, it is sufficient to create an evolutionary advantage, so the neural mechanism is subsequently evolved quickly to perfect learnin”
Neuromodulation Improves the Evolution of Forward Models (2016, background): “The plain Hebbian rule can be generalized to specific types of pre- or post-synaptic correlations [25], but here we focus on the simpler, more commonly used version in Eq.”
Evolving Plastic Neuromodulated Networks for Behavior Emergence of Autonomous Virtual Characters (2013, methodology): “For example, to encode a neuromodulated network, Soltoggio et al. (Soltoggio et al. (2007)) used the tokens NE to indicate a standard neuron, MO to indicate a modulatory neuron and TE to delimit a terminal sequence.”
Evolutionary Advantages of Neuromodulated Plasticity in Dynamic, Reward-based Scenarios (2008, methodology): “This experimental setting was chosen also by Soltoggio et al. (2007) to show that modulatory architectures could freely develop throughout evolution to achieve higher performance than in (Niv et al., 2002).”
Evaluating Topological Models of Neuromodulation in Polyworld (2014, methodology): “The model developed in this paper is similar to specific components of modeling work done by Parussel (2006) and Soltoggio et al. (2007) in addition to work done using GasNets (McHale and Husbands, 2003).”
Genetic Representation and Evolvability of Modular Neural Controllers (2010, methodology): “This implicit encoding, which has been used not only in the domain of neuroevolution [45], [46], [47], but also in the design of electronic circuits [48] and the reverse engineering of genetic regulatory networks [49], [50], shares to some degree the capability of developmental encoding to provide a”
Adaptation of Robot Behaviour through Online Evolution and Neuromodulated Learning (2012, background): “In [27], it was shown that neuromodulation allows the evolution of behaviours with a complex reinforcement learning dynamic.”
Evolving Neuromodulatory Topologies for Plasticity in Video Game Playing (2016, methodology): “[21] However, because of the ability to change the weights of certain synapses, the neuromodulation could have been responsible for making permanent changes in an agent’s movements after reaching a certain point of the environment, enabling a different kind of behavior after this point.”
Neuroevolution: from architectures to learning (2008, result): “solution during evolution, outperforming the results obtained with hand-designed value-based learning architectures [78].”
Evolving Plastic Neural Networks for Online Learning: Review and Future Directions (2012, methodology): “in 2002 [20] and has been used by several subsequent authors [22, 23, 25, 26, 28, 29].”
Using a map-based encoding to evolve plastic neural networks (2011, methodology): “Some of the early works relied on simple homeostatic rules [24], while more recent works employed more complex rules from a set of heterosynaptic primitives [4], [15], [25], [26] and introduced neuro-modulation [5], [8].”
Spectrum-Diverse Neuroevolution With Unified Neural Models (2017, background): “The results obtained from using such mechanisms are much more encouraging [35]–[39].”
Real-time Hebbian Learning from Autoencoder Features for Control Tasks (2014, background): “…and learning, neural plasticity has long captivated artificial life and related fields (Baxter, 1992; Floreano and Urzelai, 2000; Niv et al., 2002; Soltoggio et al., 2008, 2007; Soltoggio and Jones, 2009; Soltoggio and Stanley, 2012; Risi and Stanley, 2010; Risi et al., 2011; Risi and Stanley,…”
Emergent vision system of autonomous virtual characters (2016, methodology): “Soltoggio et al. proposed the use of a model of neuromodulation in T-Maze [28] and in bee foraging behavior [27].”
How novelty search escapes the deceptive trap of learning to learn (2009, unknown): “Although this work also showed that recurrent fixed-weight networks can be more effective and reliable than adaptive Hebbian controllers in some domains, previous studies [15, 18, 19] suggest that both network types reach their limits when more elaborate forms of learning are needed.”
Evolving plastic neural networks with novelty search (2010, methodology): “…hypothesis that novelty search escapes deception in adaptive domains because it is well-established from prior work (Blynel & Floreano, 2003; Dürr, Mattiussi, Soltoggio, & Floreano, 2008; Soltoggio et al., 2007, 2008) and can be adjusted to be more or less deceptive, as is done in this article.”