Neural computations underlying arbitration between model-based and model-free learning

By Sang Wan Lee, Shinsuke Shimojo, John P. O'Doherty. Neuron, 2013

592 citations recorded by the source corpus.

SUMMARY There is accumulating neural evidence to support the existence of two distinct systems for guiding action-selection in the brain, a deliberative “model-based” and a reflexive “model-free” system. However, little is known about how the brain determines which of these systems controls behavior at one moment in time. We provide evidence for an arbitration mechanism that allocates the degree of control over behavior by model-based and model-free systems as a function of the reliability of their respective predictions. We show that inferior lateral prefrontal and frontopolar cortex encode both reliability signals and the output of a comparison between those signals, implicating these regions in the arbitration process. Moreover, connectivity between these regions and model-free valuation areas is negatively modulated by the degree of model-based control in the arbitrator, suggesting that arbitration may work through modulation of the model-free valuation system when the arbitrator deems that the model-based system should drive behavior.

Subfields: Cognitive Neuroscience

View Paper: https://www.semanticscholar.org/paper/d911976f1fced41125164250f9279edf1e672871

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Citations in the corpus

Psychology of Habit. (2016, background): “In an initial dual-system RL model, Daw et al. (2005) assumed a competitive, winner-take-all mechanism in which the habit system or the goal system gained control over action, depending on which system provided the more reliable estimates of action values.”
When Does Model-Based Control Pay Off? (2016, background): “It has been hypothesized that this trade-off between accuracy and demand plays a pivotal role in the arbitration between the two strategies [38, 46 – 50], but so far direct evidence for arbitration has been sparse [51].”
The successor representation in human reinforcement learning (2016, methodology): “This is consistent with SR-based algorithms but inconsistent with either MB algorithms or accounts of different MF-MB arbitration strategies 24 following reward vs. state prediction errors.”
Metacontrol for Adaptive Imagination-Based Optimization (2017, background): “…2013), that they adapt the amount of computation they perform with their model to the demands of the task (Hamrick et al., 2015), and that they trade off between multiple strategies of varying quality (Lee et al., 2014; Lieder et al., 2014; Lieder & Grifﬁths, in revision; Kool et al., in press).”
The Power of Goal-Directed Processes in the Causation of Emotional and Other Actions (2017, unknown): “This is consistent with the parallel-competitive architecture of those standard dual-process models that assume that both processes operate in parallel and then compete to determine behavior (Daw et al., 2005; Lee et al., 2014).”
Learning with Humans in the Loop (2020, background): “In neuro- and cognitive science, the debate is about which paradigm best describes human learning [23, 37].”