Functional Requirements for Reward-Modulated Spike-Timing-Dependent Plasticity

By Nicolas Frémaux, Henning Sprekeler, Wulfram Gerstner. 2010

154 citations recorded by the source corpus.

Recent experiments have shown that spike-timing-dependent plasticity is influenced by neuromodulation. We derive theoretical conditions for successful learning of reward-related behavior for a large class of learning rules where Hebbian synaptic plasticity is conditioned on a global modulatory factor signaling reward. We show that all learning rules in this class can be separated into a term that captures the covariance of neuronal firing and reward and a second term that presents the influence of unsupervised learning. The unsupervised term, which is, in general, detrimental for reward-based learning, can be suppressed if the neuromodulatory signal encodes the difference between the reward and the expected reward—but only if the expected reward is calculated for each task and stimulus separately. If several tasks are to be learned simultaneously, the nervous system needs an internal critic that is able to predict the expected reward for arbitrary stimuli. We show that, with a critic, reward-modulated spike-timing-dependent plasticity is capable of learning motor trajectories with a temporal resolution of tens of milliseconds. The relation to temporal difference learning, the relevance of block-based learning paradigms, and the limitations of learning with a critic are discussed.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks (2017, methodology): “In contrast to previous work [24, 48, 49], we have used a deterministic surrogate gradient approach instead of the commonly used”
Backpropamine: training self-modifying neural networks with differentiable neuromodulated plasticity (2018, background): “…Hebbian plasticity (e.g. the inherent interference between the unsupervised component of Hebbian learning and reward-based modiﬁcations; Frémaux et al. 2010; Frémaux & Gerstner 2015), so as to facilitate the automatic design of efﬁcient, self-contained reinforcement learning systems.”
Neuronal Reward and Decision Signals: From Theories to Data. (2015, methodology): “Formal modeling demonstrates the suitability and power of dopamine-like reward prediction error signals in STDP learning (170, 247, 446), including prevention of run-away synaptic weights (375, 457).”
Reward-based training of recurrent neural networks for cognitive and value-based tasks (2016, methodology): “In this and other works ( Fre´maux et al., 2010 ), moreover, the prediction error was computed by algorithmically keeping track of a stimulus (task condition)-specific running average of rewards.”
Synaptic Consolidation: From Synapses to Behavioral Modeling (2015, background): “…al., 2008; Barrett et al., 2009), the state of the slow synaptic trace is normally protected against overwriting and cannot change unless a “write-now” signal is given (Crow, 1968; Carpenter and Grossberg, 1987; Izhikevich, 2007; Fre´maux et al., 2010; Fre´gnac et al., 2010; Pawlak et al., 2010).”
Stochastic variational learning in recurrent spiking networks (2014, background): “…in biology, however, certain types of non-local signals participate in the learning process, notably through neuromodulators which can convey information about the global state of the network or external information (e.g., reward or surprise) (Izhikevich, 2007; Schultz, 2008; Fremaux et al., 2010).”
A Spiking Neural Model for Stable Reinforcement of Synapses Based on Multiple Distal Rewards (2013, methodology): “Thefact that the biased learning rule, R-STDP, could learn more than one r -pattern seems to be at odds with the conclusions of Fr´emaux et al. (2010) that a biased learning rule cannot learn multiple r -patterns.”
Neuromodulated Spike-Timing-Dependent Plasticity, and Theory of Three-Factor Learning Rules (2016, methodology): “…at the junction leads to a reward or not (Arleo and Gerstner, 2000; Foster et al., 2000; Xie and Seung, 2004; Pfister et al., 2006; Baras and Meir, 2007; Florian, 2007; Di Castro et al., 2009; Sheynikhovich et al., 2009; Urbanczik and Senn, 2009; Vasilaki et al., 2009; Frémaux et al., 2010, 2013).”
Eligibility Traces and Plasticity on Behavioral Time Scales: Experimental Support of NeoHebbian Three-Factor Learning Rules (2018, methodology): “r is deﬁned as reward minus expected reward (Fr´emaux and Gerstner, 2016). In policy gradient methods and related three-factor rules, expected reward is calculated as a running average of the reward (Fremaux et al., 2010) or ﬁxed to zero by choice of reward schedule (Florian, 2007; Leg-´ enstein et ”