A distributional code for value in dopamine-based reinforcement learning

By Will Dabney, Zeb Kurth-Nelson, Naoshige Uchida, C. Starkweather, Demis Hassabis, R´emi Munos, Matthew Botvinick. Nature, 2020

418 citations recorded by the source corpus.

Since its introduction, the reward prediction error theory of dopamine has explained a wealth of empirical phenomena, providing a unifying framework for understanding the representation of reward and value in the brain1–3. According to the now canonical theory, reward predictions are represented as a single scalar quantity, which supports learning about the expectation, or mean, of stochastic outcomes. Here we propose an account of dopamine-based reinforcement learning inspired by recent artificial intelligence research on distributional reinforcement learning4–6. We hypothesized that the brain represents possible future rewards not as a single mean, but instead as a probability distribution, effectively representing multiple future outcomes simultaneously and in parallel. This idea implies a set of empirical predictions, which we tested using single-unit recordings from mouse ventral tegmental area. Our findings provide strong evidence for a neural realization of distributional reinforcement learning. Analyses of single-cell recordings from mouse ventral tegmental area are consistent with a model of reinforcement learning in which the brain represents possible future rewards not as a single mean of stochastic outcomes, as in the canonical model, but instead as a probability distribution.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience

View Paper: https://www.semanticscholar.org/paper/86a9a8cff082ddba64e00969274d4408db2b4811

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

Citations in the corpus

Distributional reinforcement learning in the brain (2020, methodology): “Distributional RL lends itself to several additional experimental predictions, which remain to be tested [8].”
Universal Off-Policy Evaluation (2021, background): “[3] instead hypothesized that the brain represents possible future rewards not as a single mean, but instead as a probability distribution, effectively representing multiple future outcomes simultaneously.”
Distributional Soft Actor-Critic: Off-Policy Reinforcement Learning for Addressing Value Estimation Errors (2020, background): “[33] found that the brain represents possible future rewards not as a single mean but instead as a probability distribution through mouse experiments.”
The Role of Executive Function in Shaping Reinforcement Learning. (2020, background): “The neural foundations of reward-dependent learning [7], and its various successors [8], have established a well-defined brain network that performs key RL computations.”
Transforming task representations to perform novel tasks (2020, methodology): “RL-like computations relate to neural activity (20, 21), and RL has driven recent artiﬁcial intelligence achievements in complex tasks like Go and StarCraft (22, 23).”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, positive): “Will et al. [201] used dopamine’s reward prediction error theory to explain rich empirical phenomena.”