Simple Plans or Sophisticated Habits? State, Transition and Learning Interactions in the Two-Step Task

By Thomas E. Akam, Rui M. Costa, Peter S. Dayan. 2015

163 citations recorded by the source corpus.

The recently developed ‘two-step’ behavioural task promises to differentiate model-based or goal-directed from model-free or habitual reinforcement learning, while generating neurophysiologically-friendly decision datasets with parametric variation of decision variables. These desirable features have prompted widespread adoption of the task. However, the signatures of model-based control can be elusive – here, we investigate model-free learning methods that, depending on the analysis strategy, can masquerade as being model-based. We first show that unadorned model-free reinforcement learning can induce correlations between action values at the start of the trial and the subsequent trial events in such a way that analysis based on comparing successive trials can lead to erroneous conclusions. We also suggest a correction to the analysis that can alleviate this problem. We then consider model-free reinforcement learning strategies based on different state representations from those envisioned by the experimenter, which generate behaviour that appears model-based under these, and also more sophisticated, analyses. The existence of such strategies is of particular relevance to the design and interpretation of animal studies using the two-step task, as extended training and a sharp contrast between good and bad options are likely to promote their use. Author Summary Planning is the use of a predictive model of the consequences of actions to guide decision making. Planning plays a critical role in human behaviour but isolating its contribution is challenging because it is complemented by control systems which learn values of actions directly from the history of reinforcement, resulting in automatized mappings from states to actions often termed habits. Our study examined a recently developed behavioural task which uses choices in a multi-step decision tree to differentiate planning from value-based control. Using simulation, we demonstrated the existence of strategies which produce behaviour that resembles planning but in fact arises as a fixed mapping from particular sorts of states to actions. These results show that when a planning problem is faced repeatedly, sophisticated automatization strategies may be developed which identify that there are in fact a limited number of relevant states of the world each with an appropriate fixed or habitual response. Understanding such strategies is important for the design and interpretation of tasks which aim to isolate the contribution of planning to behaviour. Such strategies are also of independent scientific interest as they may contribute to automatization of behaviour in complex environments.

Subfields: Artificial Intelligence, Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

Citations in the corpus

Learning to reinforcement learn (2016, background): “However, as has been remarked in very recent work, the same pattern can arise from a model-free system with an appropriate state representation (Akam et al., 2015).”
Hippocampal Contributions to Model-Based Planning and Spatial Memory (2019, unknown): “684 Neuron 102, 683–693, May 8, 2019 lightly trained humans (Akam et al., 2015; Economides et al., 2015).”
AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence (2019, background): “It can also seemingly build an internal model of the world and plan within that model to behave like model-based RL methods instead of model-free RL methods, or at least the networks it produces pass (admittedly imperfect [3]) tests designed to tell model-based methods from model-free methods [191].”
When Does Model-Based Control Pay Off? (2016, methodology): “For the model-free agent, this reflects the presence of a highly rewarding action that the model-free learner learns to approach (for a detailed analysis, see [9]).”