Dopamine reward prediction errors reflect hidden state inference across time

By C. Starkweather, Benedicte M Babayan, Naoshige Uchida, Samuel J. Gershman. 2017

193 citations recorded by the source corpus.

Midbrain dopamine neurons signal reward prediction error (RPE), or actual minus expected reward. The temporal difference (TD) learning model has been a cornerstone in understanding how dopamine RPEs could drive associative learning. Classically, TD learning imparts value to features that serially track elapsed time relative to observable stimuli. In the real world, however, sensory stimuli provide ambiguous information about the hidden state of the environment, leading to the proposal that TD learning might instead compute a value signal based on an inferred distribution of hidden states (a 'belief state'). Here we asked whether dopaminergic signaling supports a TD learning framework that operates over hidden states. We found that dopamine signaling showed a notable difference between two tasks that differed only with respect to whether reward was delivered in a deterministic manner. Our results favor an associative learning rule that combines cached values with hidden-state inference.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

1. Doina Precup
2. Irina Rish
3. Jan Drugowitsch
4. Naoshige Uchida
5. Khimya Khetarpal
6. Matthew Riemer
7. A. G. Collins

In Cognitive Neuroscience

1. Milena Rmus
2. Doina Precup
3. Irina Rish
4. Jan Drugowitsch
5. Naoshige Uchida
6. Khimya Khetarpal
7. Matthew Riemer
8. A. G. Collins

In Molecular Biology

1. Doina Precup
2. Irina Rish
3. Naoshige Uchida
4. Khimya Khetarpal
5. Matthew Riemer
6. A. G. Collins

Citations in the corpus

• Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Indeed, there has been a significant body of research trying to understand the role of the dopamine system as a reward signal for humans (Samson et al., 2010; Starkweather et al., 2017).”
• Distributional reinforcement learning in the brain (2020, background): “For example, there is strong evidence that reward expectations inferred from ambiguous state information [71,72,82] or perceptual uncertainty [83,84] modulate dopamine activity.”
• The Role of Executive Function in Shaping Reinforcement Learning. (2020, background): “The prefrontal cortex also appears to track information and uncertainty [40], which can be held in working memory to influence decision making [62] (Figure 4).”