Cognitive Control Predicts Use of Model-based Reinforcement Learning

By A. Ross Otto, Anya Skatova, Seth Madlon-Kay, Nathaniel D. Daw. 2014

184 citations recorded by the source corpus.

Abstract Accounts of decision-making and its neural substrates have long posited the operation of separate, competing valuation systems in the control of choice behavior. Recent theoretical and experimental work suggest that this classic distinction between behaviorally and neurally dissociable systems for habitual and goal-directed (or more generally, automatic and controlled) choice may arise from two computational strategies for reinforcement learning (RL), called model-free and model-based RL, but the cognitive or computational processes by which one system may dominate over the other in the control of behavior is a matter of ongoing investigation. To elucidate this question, we leverage the theoretical framework of cognitive control, demonstrating that individual differences in utilization of goal-related contextual information—in the service of overcoming habitual, stimulus-driven responses—in established cognitive control paradigms predict model-based behavior in a separate, sequential choice task. The behavioral correspondence between cognitive control and model-based RL compellingly suggests that a common set of processes may underpin the two behaviors. In particular, computational mechanisms originally proposed to underlie controlled behavior may be applicable to understanding the interactions between model-based and model-free choice behavior.

Subfields: Cognitive Neuroscience, General Decision Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In General Decision Sciences

Citations in the corpus

Model-based learning protects against forming habits (2015, result): “Consistent with either view, the model-based index has repeatedly proved to be more sensitive to manipulations and individual differences (Otto, Gershman, et al., 2013; Otto, Raio, et al., 2013; Otto et al., 2015; Voon et al., 2014; Wunderlich, Smittenaar, & Dolan, 2012).”
When Does Model-Based Control Pay Off? (2016, background): “Furthermore, the degree of model-free control on this task is predicted by individual difference measures such as working memory capacity [23], cognitive control ability [24], processing speed [29], age [20, 31], extraversion [30], and even psychiatric pathology [11, 33, 34].”
Psychology of Habit. (2016, background): “…in individual-difference paradigms, older adults and those with lower cognitive-control abilities were less able to leverage higher-order goal representations for model-based responding in order to overcome habitual, model-free solutions to a variety of tasks (de Wit et al. 2014, Otto et al. 2015).”
Rule abstraction, model-based choice, and cognitive reflection (2016, background): “However, model-based choice is predicted by the use of proactive strategy in cognitive control (Otto et al., 2015), and these control strategies can be modified with some”