When Does Model-Based Control Pay Off?

By Wouter Kool, Fiery Cushman, Samuel J. Gershman. 2016

195 citations recorded by the source corpus.

Many accounts of decision making and reinforcement learning posit the existence of two distinct systems that control choice: a fast, automatic system and a slow, deliberative system. Recent research formalizes this distinction by mapping these systems to “model-free” and “model-based” strategies in reinforcement learning. Model-free strategies are computationally cheap, but sometimes inaccurate, because action values can be accessed by inspecting a look-up table constructed through trial-and-error. In contrast, model-based strategies compute action values through planning in a causal model of the environment, which is more accurate but also more cognitively demanding. It is assumed that this trade-off between accuracy and computational demand plays an important role in the arbitration between the two strategies, but we show that the hallmark task for dissociating model-free and model-based strategies, as well as several related variants, do not embody such a trade-off. We describe five factors that reduce the effectiveness of the model-based strategy on these tasks by reducing its accuracy in estimating reward outcomes and decreasing the importance of its choices. Based on these observations, we describe a version of the task that formally and empirically obtains an accuracy-demand trade-off between model-free and model-based strategies. Moreover, we show that human participants spontaneously increase their reliance on model-based control on this task, compared to the original paradigm. Our novel task and our computational analyses may prove important in subsequent empirical investigations of how humans balance accuracy and demand.

Subfields: Cognitive Neuroscience, Experimental and Cognitive Psychology, General Decision Sciences

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Experimental and Cognitive Psychology

In General Decision Sciences

Citations in the corpus

Learning to reinforcement learn (2016, methodology): “We used a modiﬁed version of the two-step task, designed to bolster the utility of model-based over model-free control (see Kool et al., 2016).”
Prioritized memory access explains planning and hippocampal replay (2017, background): “…suggesting they selectively engage or omit such computations as appropriate to the circumstances (via trading off two algorithms for value estimation, known as model-based and model-free) (Daw et al., 2005; Gillan et al., 2016; Keramati et al., 2011, 2016; Kool et al., 2016; Pezzulo et al., 2013).”