States versus Rewards: Dissociable neural prediction error signals underlying model-based and model-free reinforcement learning

By Jan Gläscher, Nathaniel D. Daw, Peter S. Dayan, John P. O'Doherty. 2010

1,168 citations recorded by the source corpus.

Reinforcement learning (RL) uses sequential experience with situations (“states”) and outcomes to assess actions. Whereas model-free RL uses this experience directly, in the form of a reward prediction error (RPE), model-based RL uses it indirectly, building a model of the state transition and outcome structure of the environment, and evaluating actions by searching this model. A state prediction error (SPE) plays a central role, reporting discrepancies between the current model and the observed state transitions. Using functional magnetic resonance imaging in humans solving a probabilistic Markov decision task we found the neural signature of an SPE in the intraparietal sulcus and lateral prefrontal cortex, in addition to the previously well-characterized RPE in the ventral striatum. This finding supports the existence of two unique forms of learning signal in humans, which may form the basis of distinct computational strategies for guiding behavior.

Subfields: Cognitive Neuroscience, Reinforcement Learning, Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Reinforcement Learning

In Neuroscience

Citations in the corpus

Continuous control with deep reinforcement learning (2015, background): “that a robust model-free approach may be an important component of larger systems which may attack these limitations [24].”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Research suggests that a combination of model-free and model-based computations are performed in the human brain (Gläscher et al., 2010; Daw et al., 2011; Doll et al., 2012).”
Metacontrol for Adaptive Imagination-Based Optimization (2017, background): “Evidence suggests that humans rely on rich generative models of the world for planning (Gl¨ascher et al., 2010), control (Wolpert & Kawato, 1998), and reasoning (Hegarty, 2004; Johnson-Laird, 2010; Battaglia et al., 2013), that they adapt the amount of computation they perform with their model to…”
Explicit and Implicit Processes Constitute the Fast and Slow Processes of Sensorimotor Learning (2015, background): “…on dissociating various operations involved in learning and memory, such as model-based and model-free reinforcement learning (Daw et al., 2005; Gläscher et al., 2010), intentional and automatic forms of memory (Jacoby, 1991), and rule-based versus information-integration category learning…”
When Does Model-Based Control Pay Off? (2016, methodology): “Behavior on the Daw two-step task can be modeled using an established dual-system reinforcement-learning model [7, 8, 40].”
A map of abstract relational knowledge in the human hippocampal–entorhinal cortex (2017, background): “Indeed, we note that neural signals can be recorded in frontal and parietal cortices, reflecting the ‘state-prediction errors’ that ensue when predicted state relationships are breached during behavioural control (Gläscher et al., 2010).”
Learning with Humans in the Loop (2020, background): “In neuro- and cognitive science, the debate is about which paradigm best describes human learning [23, 37].”
Offline replay supports planning in human reinforcement learning (2018, background): “Similarly, there is evidence that neural replay is sensitive to the sign of prediction errors (Gläscher et al., 2010).”
Neural computations underlying arbitration between model-based and model-free learning (2013, positive): “Although there is considerable evidence for the existence of those computations[2,3], a fundamental question that has”
Revealing Neurocomputational Mechanisms of Reinforcement Learning and Decision-Making With the hBayesDM Package (2016, positive): “…particularly popular in cognitive neuroscience (e.g., Ahn, Krawits, et al., 2011; Behrens, Woolrich, Walton, & Rushworth, 2007; Daw et al., 2006; Gläscher, Daw, Dayan, & Doherty, 2010; Gläscher et al., 2009; Hampton et al., 2006; Iglesias et al., 2013; Kable & Glimcher, 2007; O’Doherty,…”
Insights from the application of computational neuroimaging to social neuroscience. (2013, background): “…] bears a striking resemblance both in terms of its neuroanatomical substrates and the nature of the computational learning signals involved to that of a ‘state prediction error’ [42], a key computational signal responsible for updating of the model in model-based reinforcement learning [42,43].”
Disorders of compulsivity: a common bias towards learning habits (2014, positive): “BOLD activity in various cortical regions covaries with aspects of model-based learning in healthy volunteers: for instance, the state prediction error, or the discrepancy between the observed and expected state transition is represented in the lateral prefrontal cortex and intraparietal sulcus (9).”
Thermodynamics as a theory of decision-making with information-processing costs (2012, background): “ntaminated with additive noise following an extreme value distribution. While stochastic choice models are generally accepted to account for human choices better than their deterministic counterparts [77, 78, 79], they have also been strongly criticized, especially for a property known as independen”