Evolution of Reinforcement Learning in Uncertain Environments: A Simple Explanation for Complex Foraging Behaviors

By Yael Niv, Daphna Joel, Isaac Meilijson, Eytan Ruppin. 2002

193 citations recorded by the source corpus.

Subfields: Cellular and Molecular Neuroscience, Ecology, Evolution, Behavior and Systematics, Genetics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Ecology, Evolution, Behavior and Systematics

In Genetics

Citations in the corpus

Born to Learn: the Inspiration, Progress, and Future of Evolved Plastic Artificial Neural Networks (2017, background): “RL in neural networks (Pennartz, 1997) makes use of robotic or agent-environment problems with a reward signal that affects plasticity in the network (Lin, 1993; Niv et al., 2002; Soltoggio and Stanley, 2012; Auerbach et al., 2014; Pugh et al., 2014).”
Using a map-based encoding to evolve plastic neural networks (2011, background): “Hence, all the successful previous experiments involved from 1 [15], [27] to 3 [30] outputs and at most 8 inputs.”
Neural Modularity Helps Organisms Evolve to Learn New Skills without Forgetting Old Skills (2015, methodology): “This type of plasticity-controlling neuromodulation has been successfully applied when evolving neural networks that solve reinforcement learning problems [25, 46], and a comparison found that evolution was able to solve more complex tasks with neuromodulated Hebbian learning than with Hebbian learn”
Guided self-organization in indirectly encoded and evolving topographic maps (2014, background): “While neuroevolution (NE), i.e. evolving artificial neural networks (ANNs) through evolutionary algorithms, has produced significant results in a variety of different domains [7, 28, 33], the ANNs evolved through NE still often do not include some of the important features seen in natural brains.”
Evolving plastic neural networks with novelty search (2010, background): “The generalized Hebbian plasticity rule (Niv et al., 2002) takes the following form:”
The evolution of a generalized neural learning rule (2016, background): “Other examples of evolving learning appeared, many of which used a parameterized Hebbian rule as an update function [6]–[8].”
Novelty of behaviour as a basis for the neuro-evolution of operant reward learning (2009, methodology): “environment. Although optimality in control is a coveted feature, there are situations where optimal control policies are not defined, for example in a new environment, or when the outcome of certain actions cannot be known in advance [ 10 ].”
Reinforcement learning: the good, the bad and the ugly. (2008, unknown): “Although nonlinear utilities for rewards will affect both types of controllers, the effects of risk through sampling biases [114] or through a learning rule that is asymmetrically sensitive to positive and negative errors [115] depend on the specific implementation of learning in a model-free or mod”
Evolving Plastic Neural Networks for Online Learning: Review and Future Directions (2012, methodology): “This requires producing environments that are different in some way for each generation and/or change in some way during a fitness evaluation [4, 20, 25, 27].”
A unified approach to evolving plasticity and neural geometry (2012, background): “The generalized Hebbian plasticity rule [18] takes the form: ∆w ji = η · [Ao j o i + Bo j + Co i + D], where η is the learning rate, o j and o i are the activation levels of the presynaptic and postsynaptic neurons and A–”
From modulated Hebbian plasticity to simple behavior learning through noise and weight saturation (2012, background): “The dynamic stochastic reward contingencies in this last experiment are often found in real scenarios (Montague et al., 1995; Niv et al., 2002).”
Evolutionary and Computational Advantages of Neuromodulated Plasticity (2008, methodology): “In this thesis, the simulated bee and the uncertain environment in (Niv et al., 2002) were reproduced.”
Lessons for artificial intelligence from the study of natural stupidity (2019, background): “It is also exhibited by non-human decision makers, such as foraging honey bees 46 , and has been shown to emerge even in an optimal decision-making agent 47 .”
Evolutionary Advantages of Neuromodulated Plasticity in Dynamic, Reward-based Scenarios (2008, methodology): “Equation 4 has been used in previous studies of neuromodulation ( Niv et al., 2002; Soltoggio et al., 2007).”
Neural Plasticity and Minimal Topologies for Reward-Based Learning (2008, methodology): “In contrast to previous studies [9, 10], the results indicate that complex learning problems in uncertain stochastic environments can be solved with simple homosy-naptic plasticity rules and minimal architectures.”