Spatio-Temporal Credit Assignment in Neuronal Population Learning

By Johannes Friedrich, Robert Urbanczik, Walter Senn. 2011

66 citations recorded by the source corpus.

In learning from trial and error, animals need to relate behavioral decisions to environmental reinforcement even though it may be difficult to assign credit to a particular decision when outcomes are uncertain or subject to delays. When considering the biophysical basis of learning, the credit-assignment problem is compounded because the behavioral decisions themselves result from the spatio-temporal aggregation of many synaptic releases. We present a model of plasticity induction for reinforcement learning in a population of leaky integrate and fire neurons which is based on a cascade of synaptic memory traces. Each synaptic cascade correlates presynaptic input first with postsynaptic events, next with the behavioral decisions and finally with external reinforcement. For operant conditioning, learning succeeds even when reinforcement is delivered with a delay so large that temporal contiguity between decision and pertinent reward is lost due to intervening decisions which are themselves subject to delayed reinforcement. This shows that the model provides a viable mechanism for temporal credit assignment. Further, learning speeds up with increasing population size, so the plasticity cascade simultaneously addresses the spatial problem of assigning credit to synapses in different population neurons. Simulations on other tasks, such as sequential decision making, serve to contrast the performance of the proposed scheme to that of temporal difference-based learning. We argue that, due to their comparative robustness, synaptic plasticity cascades are attractive basic models of reinforcement learning in the brain.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

Incorporating Learnable Membrane Time Constant to Enhance Learning of Spiking Neural Networks (2020, methodology): “Existing approaches exploited the self-organization principle [56, 11, 29], and STDP-based expectation-maximization algorithm [43, 17].”
Neuromodulated Spike-Timing-Dependent Plasticity, and Theory of Three-Factor Learning Rules (2016, methodology): “Traditional studies of dopamine-modulated plasticity in cortico-striatal synapses have also relied on “rate-based” protocols, where no particular attention is paid to the relative timing of pre- and postsynaptic spikes (see Reynolds and Wickens, 2002 and Jay, 2003, for reviews).”
A Spiking Neural Model for Stable Reinforcement of Synapses Based on Multiple Distal Rewards (2013, methodology): “Friedrich, Urbanczik, and Senn (2011) also solve the distal reward problem using a variant of the R-Max algorithm.”
Dendritic cortical microcircuits approximate the backpropagation algorithm (2018, background): “This is often referred to as the synaptic credit assignment problem (Rumelhart et al., 1986; Sutton and Barto, 1998; Roelfsema and van Ooyen, 2005; Friedrich et al., 2011; Bengio, 2014; Lee et al., 2015; Roelfsema and Holtmaat, 2018), for which the backprop algorithm provides an elegant solution.”