Dopamine, uncertainty and TD learning

By F. Sluyter, Laure Jamot, J. Bertholet, Wim E. Crusio. Behavioral and Brain Functions, 2005

611 citations recorded by the source corpus.

Substantial evidence suggests that the phasic activities of dopaminergic neurons in the primate midbrain represent a temporal difference (TD) error in predictions of future reward, with increases above and decreases below baseline consequent on positive and negative prediction errors, respectively. However, dopamine cells have very low baseline activity, which implies that the representation of these two sorts of error is asymmetric. We explore the implications of this seemingly innocuous asymmetry for the interpretation of dopaminergic firing patterns in experiments with probabilistic rewards which bring about persistent prediction errors. In particular, we show that when averaging the non-stationary prediction errors across trials, a ramping in the activity of the dopamine neurons should be apparent, whose magnitude is dependent on the learning rate. This exact phenomenon was observed in a recent experiment, though being interpreted there in antipodal terms as a within-trial encoding of uncertainty.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience

View Paper: https://www.semanticscholar.org/paper/8831b164633e96fe2b4e9f9f223db7be82044d66

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

Citations in the corpus

European consensus statement on diagnosis and treatment of adult ADHD: The European Network Adult ADHD (2010, neutral): “4 in adults [32,293], a duration of action of 24 hours, and no abuse potential [294].”
Neural systems implicated in delayed and probabilistic reinforcement (2006, background): “DA neurons that innervate the Acb may fire in a manner related to reward probability (Fiorillo et al., 2003, 2005; Niv et al., 2005; Tobler et al., 2005) and the midbrain, the site of the cell bodies of these neurons, responds to stimulus uncertainty in humans (Aron et al.”
Reinforcement learning in the brain (2009, positive): “3a, b) and firing patterns in tasks involving probabilistic rewards are in accord with a constantly back-propagating error signal (Niv et al., 2005, Fig.”
The basal ganglia: learning new tricks and loving it. (2005, neutral): “[20]).”
Neuronal Reward and Decision Signals: From Theories to Data. (2015, methodology): “As the backpropagating activation would be small and dispersed, it would appear as ramp only when averaging across multiple trials (183, 388).”
Habits, rituals, and the evaluative brain. (2008, positive): “…et al. 2003, Tobler et al. 2005, Waelti et al. 2001), that they are specialized to respond in relation to positive but not aversive reinforcements, and that they may code uncertainty (Fiorillo et al. 2003, Hsu et al. 2005, Niv et al. 2005, Ungless et al. 2004) or salience (Redgrave & Gurney 2006).”
Neural differentiation of expected reward and risk in human subcortical structures. (2006, positive): “Under TD learning, activation for expected reward should be significantly correlated with this indicator variable (while activation for risk could be explained as an effect of backpropagation of prediction errors [Fiorillo et al., 2005; Niv et al., 2005]).”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “For example, there is significant evidence in the literature that human learning is either uncertainty-aware or Bayesian in nature (Niv et al., 2005; Fleming and Daw, 2017; Babayan et al., 2018; Lowet et al., 2020).”
The free-energy principle: a unified brain theory? (2010, neutral): “Conversely, it may provide a framework in which current debates could be resolved, for example whether dopamine encodes reward prediction error or surpris”
The computational neurobiology of learning and reward. (2006, background): “, variability) co-located with the prediction error signal, but an alternative explanation is that the ramp is a side effect of the standard TD error signal, related to the anticipatory nature of the learned predictions [47].”
Safe models for risky decisions (2017, background): “, 2002), attention-deficit-hyperactivity disorder (Agay, Yechiam, Carmel, & Levkovitz, 2010; Toplak et al., 2005), obsessive-compulsive disorder (Cavedini, Riboldi, D’Annucci, et al.”