Solving the Distal Reward Problem with Rare Correlations

By Andrea Soltoggio, Jochen J. Steil. 2013

33 citations recorded by the source corpus.

Abstract In the course of trial-and-error learning, the results of actions, manifested as rewards or punishments, occur often seconds after the actions that caused them. How can a reward be associated with an earlier action when the neural activity that caused that action is no longer present in the network? This problem is referred to as the distal reward problem. A recent computational study proposes a solution using modulated plasticity with spiking neurons and argues that precise firing patterns in the millisecond range are essential for such a solution. In contrast, the study reported in this letter shows that it is the rarity of correlating neural activity, and not the spike timing, that allows the network to solve the distal reward problem. In this study, rare correlations are detected in a standard rate-based computational model by means of a threshold-augmented Hebbian rule. The novel modulated plasticity rule allows a randomly connected network to learn in classical and instrumental conditioning scenarios with delayed rewards. The rarity of correlations is shown to be a pivotal factor in the learning and in handling various delays of the reward. This study additionally suggests the hypothesis that short-term synaptic plasticity may implement eligibility traces and thereby serve as a selection mechanism in promoting candidate synapses for long-term storage.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

In Electrical and Electronic Engineering

Citations in the corpus

From modulated Hebbian plasticity to simple behavior learning through noise and weight saturation (2012, background): “Other neural algorithms that use eligibility traces target those problems (Izhikevich, 2007; Soltoggio and Steil, 2012).”
Rare Neural Correlations Implement Robotic Conditioning with Delayed Rewards and Disturbances (2013, background): “To preserve the selectivity of the RCHP rule, longer-lastingtracesmustbecompensatedwithalowerrateof production, i.e., they must be generated even more parsimoniously.”
Real-time Hebbian Learning from Autoencoder Features for Control Tasks (2014, background): “Furthermore, as the state of the art in Hebbian learning advances, such as by addressing distal rewards (Soltoggio and Steil, 2013), the RAAHN benefits from the advancing capabilities as well.”
Learning the rules of a game: Neural conditioning in human-robot interaction with delayed rewards (2013, methodology): “Relatively few studies focus on the neural mechanisms that bridge the temporal gap between sequences of cues, actions and rewards [15], [16], [17].”