The ubiquity of model-based reinforcement learning

By Bradley B. Doll, Dylan A. Simon, Nathaniel D. Daw. 2012

393 citations recorded by the source corpus.

The reward prediction error theory of dopamine function has enjoyed great success in the neuroscience of learning and decision-making. This theory is derived from model-free reinforcement learning in which choices are made simply on the basis of previously realized rewards. Recently, attention has turned to correlates of more flexible, albeit computationally complex, model-based methods in the brain. These methods are distinguished from model-free learning by their evaluation of candidate actions using expected future outcomes according to a world model. Puzzlingly, signatures from these computations seem to be pervasive in the very same regions previously thought to support model-free learning. Here, we review recent behavioral and neural evidence about these two systems, in attempt to reconcile their enigmatic cohabitation in the brain.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Research suggests that a combination of model-free and model-based computations are performed in the human brain (Gläscher, Daw, Dayan, and O’Doherty, 2010; Daw, Gershman, Seymour, Dayan, and Dolan, 2011; Doll, Simon, and Daw, 2012).”
The minority report: some common assumptions to reconsider in the modelling of the brain and behaviour (2016, methodology): “In model-based RL (Doll, Simon, & Daw, 2012), the agent builds and maintains a model or representation of the environment, which is used to predict the reward that would ensue from various actions, given the present state of affairs of the agent and of its environment.”
Psychology of Habit. (2016, methodology): “Given the profusion of direct and indirect connections between neural circuits underlying goal-directed and habitual (model-free) behaviors, goals can have a biasing influence on habit formation (Doll et al. 2012).”
Learning to Predict Without Looking Ahead: World Models Without Forward Prediction (2019, background): “Much of the motivation of model-based reinforcement learning (RL) derives from the potential utility of learned models for downstream tasks, like prediction [13, 15], planning [1, 34, 39, 40, 42, 63], and counterfactual reasoning [9, 27].”
Model-based predictions for dopamine (2017, unknown): “That dopamine prediction errors may reﬂect this computed-on-the-ﬂy value is also consistent with accumulating evidence from fMRI showing that prediction error signals include model-based information and that model-based decisions are sensitive to striatal dopamine [43 ,44,45].”
Goals and Habits in the Brain (2013, unknown): “Various new tasks have also shed light on model-based and model-free systems (Doll et al., 2012).”
Model-based hierarchical reinforcement learning and human action control (2014, background): “However, attention has subsequently expanded to consider the potential relevance of model-based RL [4,5].”
Model-Based and Model-Free Pavlovian Reward Learning: Revaluation, Revision and Revelation (2014, background): “…are also under active investigation, at in an instrumental context, from viewpoints both theoretical (Dayan, 1993; Dezfouli & Balleine, 2012; Doll et al., 2012; Keramati, Dezfouli, & Piray, 2011; Pezzulo, Rigoli, & Chersi, 2013; Sutton & Barto, 1998) and empirical (Daw et al., 2011;…”