Belief state representation in the dopamine system

By Benedicte M Babayan, Naoshige Uchida, Samuel J. Gershman. 2018

102 citations recorded by the source corpus.

Learning to predict future outcomes is critical for driving appropriate behaviors. Reinforcement learning (RL) models have successfully accounted for such learning, relying on reward prediction errors (RPEs) signaled by midbrain dopamine neurons. It has been proposed that when sensory data provide only ambiguous information about which state an animal is in, it can predict reward based on a set of probabilities assigned to hypothetical states (called the belief state). Here we examine how dopamine RPEs and subsequent learning are regulated under state uncertainty. Mice are first trained in a task with two potential states defined by different reward amounts. During testing, intermediate-sized rewards are given in rare trials. Dopamine activity is a non-monotonic function of reward size, consistent with RL models operating on belief states. Furthermore, the magnitude of dopamine responses quantitatively predicts changes in behavior. These results establish the critical role of state inference in RL. Dopamine neurons encode reward prediction errors (RPE) that report the mismatch between expected reward and outcome for a given state. Here the authors report that when there is uncertainty about the current state, RPEs are calculated on the probabilistic representation of the current state or belief state.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

In Molecular Biology

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “For example, there is significant evidence in the literature that human learning is either uncertainty-aware or Bayesian in nature (Niv et al., 2005; Fleming and Daw, 2017; Babayan et al., 2018; Lowet et al., 2020).”
Deep Variational Reinforcement Learning for POMDPs (2018, background): “There is some evidence from recent experiments on the dopamine system in mice (Babayan et al., 2018) showing that their response to ambiguous information is consistent with a theory operating on belief states.”
The Role of Executive Function in Shaping Reinforcement Learning. (2020, background): “Current Opinion in Behavioral Sciences 2021, 38:66–73 states [38 ,39].”
Distributional reinforcement learning in the brain (2020, background): “For example, there is strong evidence that reward expectations inferred from ambiguous state information [71,72,82] or perceptual uncertainty [83,84] modulate dopamine activity.”