Model-based hierarchical reinforcement learning and human action control

By Matthew Botvinick, Ari Weinstein. 2014

161 citations recorded by the source corpus.

Recent work has reawakened interest in goal-directed or ‘model-based’ choice, where decisions are based on prospective evaluation of potential action outcomes. Concurrently, there has been growing attention to the role of hierarchy in decision-making and action control. We focus here on the intersection between these two areas of interest, considering the topic of hierarchical model-based control. To characterize this form of action control, we draw on the computational framework of hierarchical reinforcement learning, using this to interpret recent empirical findings. The resulting picture reveals how hierarchical model-based mechanisms might play a special and pivotal role in human decision-making, dramatically extending the scope and complexity of human behaviour.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Management Science and Operations Research

Citations in the corpus

Deep Successor Reinforcement Learning (2016, methodology): “Other work (see for instance, [2, 4]) have also discussed utilizing the SR for subgoal and option discovery.”
Toward an Integration of Deep Learning and Neuroscience (2016, background): “Subroutines in such processes could be derived by a process of chunking sequences of actions into single actions (Graybiel, 1998; Botvinick and Weinstein, 2014).”
The successor representation in human reinforcement learning (2016, methodology): “Using these updated representations, we compute V(s) at the end of phase 1 and again at the end of phase 2 using equation (2) for the SR learner and equation (3) for the MB learner.”
Learning Structures: Predictive Representations, Replay, and Generalization (2020, background): “It has also been shown that SR helps discover subgoals for planning in hierarchical reinforcement learning (HRL) problems [36].”
Offline replay supports planning in human reinforcement learning (2018, background): “Caching of multi-step predictive representations during replay can in turn enable generalization (Botvinick and Weinstein, 2014), faster planning especially under time pressure, and sub-goal discovery (Beckmann et al.”
Psychology of Habit. (2016, background): “Theories of hierarchical RL show that it can be advantageous to concatenate individual actions and treat the sequence as a single response unit or chunk (Botvinick & Weinstein 2014) because the faster responding enabled by chunking can lead to greater average reward (Keramati et al. 2011).”