Speed/Accuracy Trade-Off between the Habitual and the Goal-Directed Processes

By Mohammad Mahdi Keramati, Amir Dezfouli, Payam Piray. 2011

373 citations recorded by the source corpus.

Instrumental responses are hypothesized to be of two kinds: habitual and goal-directed, mediated by the sensorimotor and the associative cortico-basal ganglia circuits, respectively. The existence of the two heterogeneous associative learning mechanisms can be hypothesized to arise from the comparative advantages that they have at different stages of learning. In this paper, we assume that the goal-directed system is behaviourally flexible, but slow in choice selection. The habitual system, in contrast, is fast in responding, but inflexible in adapting its behavioural strategy to new conditions. Based on these assumptions and using the computational theory of reinforcement learning, we propose a normative model for arbitration between the two processes that makes an approximately optimal balance between search-time and accuracy in decision making. Behaviourally, the model can explain experimental evidence on behavioural sensitivity to outcome at the early stages of learning, but insensitivity at the later stages. It also explains that when two choices with equal incentive values are available concurrently, the behaviour remains outcome-sensitive, even after extensive training. Moreover, the model can explain choice reaction time variations during the course of learning, as well as the experimental observation that as the number of choices increases, the reaction time also increases. Neurobiologically, by assuming that phasic and tonic activities of midbrain dopamine neurons carry the reward prediction error and the average reward signals used by the model, respectively, the model predicts that whereas phasic dopamine indirectly affects behaviour through reinforcing stimulus-response associations, tonic dopamine can directly affect behaviour through manipulating the competition between the habitual and the goal-directed systems and thus, affect reaction time.

Subfields: Cellular and Molecular Neuroscience, Cognitive Neuroscience, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cellular and Molecular Neuroscience

In Cognitive Neuroscience

In Molecular Biology

Citations in the corpus

Perception, Action, and Utility: The Tangled Skein (2012, methodology): “As suggested by Daw et al. (2005) and Keramati et al. (2011), which circumstances elicit which approach can be explained by a sort of meta-optimization over the costs (e.g. extra computation) of maintaining the full representation relative to its benefits (better statistical accuracy).”
Analogues of mental simulation and imagination in deep learning (2019, positive): “An active area of research has investigated the overall choice of whether (and how much) to plan [79,80], treating this choice as a speedaccuracy trade-off and inspiring similarly adaptive approaches in MBDRL [64].”
Multiple Systems for Value Learning (2014, background): “Formal models extend this reasoning (Daw et al., 2005; Keramati et al., 2011; Simon and Daw, 2011a) to analyze under what circumstances the computational costs of model-based RL (for instance, in the brain, the opportunity cost of time and the caloric cost of firing neurons) are likely to be…”
Hippocampal replays under the scrutiny of reinforcement learning models. (2018, methodology): “In contrast, in MB-RL, retrieving the Q values requires some iterative tree search process where one looks into the future through the model to estimate the possible long-term consequences of immediate actions (Daw et al. 2005; Keramati et al. 2011; Lesaint et al. 2014).”
Cognitive Control Predicts Use of Model-based Reinforcement Learning (2014, background): “…treatments of how the brain balances the costs and benefits of model-based control in choosing an RL strategy (Pezzulo, Rigoli, & Chersi, 2013; Keramati et al., 2011; Daw et al., 2005) can potentially be applied to understanding the costs and benefits of cognitive control more generally.”
Environmental statistics and the trade-off between model-based and TD learning in humans (2011, background): “However, the more relevant measure is how these uncertainties translate into values [18].”
Model-Based Reasoning in Humans Becomes Automatic with Training (2015, background): “There may also be resource requirements for maintaining belief distributions over meta-parameters, such as whether the task structure changes or new fractals appear, what appropriate learning rates are, when modelbased reasoning should be deployed [1,30] and how attentional resources should be alloc”
Integrating memories to guide decisions (2015, background): “, better chance of gaining rewards) of prospection at decision time, a tradeoff which will vary depending on issues like the amount of time pressure or training [7,52,53].”
Self-Evaluation of Decision-Making: A General Bayesian Framework for Metacognitive Computation (2017, background): “, 2009), whether a subsequent step in a chain of decisions should be initiated (Dehaene & Sigman, 2012), whether to make the task easier by offloading intentions into the environment (Gilbert, 2015), or more generally when it is advantageous to deliberate (Keramati et al., 2011) or engage cognitive ”