Solving the distal reward problem through linkage of STDP and dopamine signaling

By Eugene M. Izhikevich. 2007

748 citations recorded by the source corpus.

In Pavlovian and instrumental conditioning, reward typically comes seconds after reward-triggering actions, creating an explanatory conundrum known as "distal reward problem": How does the brain know what firing patterns of what neurons are responsible for the reward if 1) the patterns are no longer there when the reward arrives and 2) all neurons and synapses are active during the waiting period to the reward? Here, we show how the conundrum is resolved by a model network of cortical spiking neurons with spike-timing-dependent plasticity (STDP) modulated by dopamine (DA). Although STDP is triggered by nearly coincident firing patterns on a millisecond timescale, slow kinetics of subsequent synaptic plasticity is sensitive to changes in the extracellular DA concentration during the critical period of a few seconds. Random firings during the waiting period to the reward do not affect STDP and hence make the network insensitive to the ongoing activity-the key feature that distinguishes our approach from previous theoretical studies, which implicitly assume that the network be quiet during the waiting period or that the patterns be preserved until the reward arrives. This study emphasizes the importance of precise firing patterns in brain dynamics and suggests how a global diffusive reinforcement signal in the form of extracellular DA can selectively influence the right synapses at the right time.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

Citations in the corpus

Born to Learn: the Inspiration, Progress, and Future of Evolved Plastic Artificial Neural Networks (2017, background): “Finally, plasticity rules for spiking neural networks (Maass and Bishop, 2001) aim to demonstrate unique learning mechanisms that emerge from spiking dynamics (Izhikevich, 2007), as well as model biological synaptic plasticity (Gerstner and Kistler, 2002b).”
Rare Neural Correlations Implement Robotic Conditioning with Delayed Rewards and Disturbances (2013, background): “In fact, as opposed to Izhikevich (2007) and Soltoggio and Steil (2013), in which stimuli were brief and impulse-like recurrent loops, might induce self-sustained activity, an unwanted regime in which neural dynamics do not .”
Supervised Learning in Spiking Neural Networks with ReSuMe: Sequence Learning, Classification, and Spike Shifting (2010, background): “However, direct implementation of the recently proposed concepts addressing this issue (see, e.g., Izhikevich, 2007) in ReSuMe is impossible, and new mechanisms need to be investigated.”
Computational models of long term plasticity and memory (2017, neutral): “The role of STDP in learning has also been investigated in many computational studies (see e.g. [Gerstner et al., 1996, Legenstein et al., 2005b, Gutig and Sompolinsky, 2006, Izhikevich, 2007, Legenstein et al., 2008, Nessler et al., 2013, Pecevski and Maass, 2016]).”
From modulated Hebbian plasticity to simple behavior learning through noise and weight saturation (2012, background): “One promising computational aspect of modulation is the possibility of increasing, decreasing or inverting the strength and sign of plasticity (Abbott, 1990; Montague et al., 1996; Florian, 2007; Porr and Wörgötter, 2007; Izhikevich, 2007; Pfeiffer et al., 2010), making neuromodulation particularly ”
Reinforcement Learning on Slow Features of High-Dimensional Input Streams (2010, background): “Such update rules are either purely phenomenological [53,55] or derived from a reward-maximization principle [47– 51].”
Gain in sensitivity and loss in temporal contrast of STDP by dopaminergic modulation at hippocampal synapses (2009, background): “Indeed, various rules of modulated Hebbian learning have been suggested and implemented in computational models (53, 54).”
Evolvable Neuronal Paths: A Novel Basis for Information and Search in the Brain (2011, methodology): “Synapses are modified by STDP via eligibility traces modulated by DA reward, as in Izhikevich’s paper [47] except that eligibility traces decay 10 times faster and reward decays 4 times faster than in the original paper, thus increasing the specificity of reward.”
SuperSpike: Supervised Learning in Multilayer Spiking Neural Networks (2017, background): “Before we move on to testing this learning rule, we first state a few of its noteworthy properties: (i) it has a Hebbian term which combines pre- and postsynaptic activity in a multiplicative manner, (ii) the learning rule is voltage-based, (iii) it is a nonlinear Hebbian rule due to the occurrence ”
Introduction to spiking neural networks: Information processing, learning and applications. (2011, methodology): “Based on these observations several models for reinforcement learning in spiking neural networks have been proposed (Florian 2005, Baras and Meir 2007, Farries and Fairhall 2007, Florian 2007, Izhikevich 2007, Vasilaki et al. 2009).”
Computational models of reinforcement learning: the role of dopamine as a reward signal (2010, background): “Reproduced with permission from Izhikevich (2007) Cogn Neurodyn (2010) 4:91–105 97 of DA on corticostriatal synaptic plasticity (Fig 6 , reviewed in Reynolds and Wickens 2002).”
Learning Spike-Based Population Codes by Reward and Population Feedback (2010, background): “…the agents seems far-fetched, initial work on reinforcement learning in spiking neurons treated the synapses as agents (Seung, 2003), whereas in later work, the agents were assumed to be the neurons (Fiete & Seung, 2006; Pﬁster, Toyoizumi, Barber, & Gerstner, 2006; Florian, 2007; Izhikevich, 2007).”
Hierarchical Error Representation: A Computational Model of Anterior Cingulate and Dorsolateral Prefrontal Cortex (2015, background): “Alternately, an eligibility trace may be realized by a transient increase in synaptic plasticity following the activation of a neuron, which decays to baseline over time (Izhikevich, 2007)”