Hierarchically organized behavior and its neural foundations: A reinforcement-learning perspective

By Matthew Botvinick, Yael Niv, Andrew Barto. 2008

646 citations recorded by the source corpus.

Research on human and animal behavior has long emphasized its hierarchical structure — the divisibility of ongoing behavior into discrete tasks, which are comprised of subtask sequences, which in turn are built of simple actions. The hierarchical structure of behavior has also been of enduring interest within neuroscience, where it has been widely considered to reflect prefrontal cortical functions. In this paper, we reexamine behavioral hierarchy and its neural substrates from the point of view of recent developments in computational reinforcement learning. Specifically, we consider a set of approaches known collectively as hierarchical reinforcement learning, which extend the reinforcement learning paradigm by allowing the learning agent to aggregate actions into reusable subroutines or skills. A close look at the components of hierarchical reinforcement learning suggests how they might map onto neural structures, in particular regions within the dorsolateral and orbital prefrontal cortex. It also suggests specific ways in which hierarchical reinforcement learning might provide a complement to existing psychological models of hierarchically structured behavior. A particularly important question that hierarchical reinforcement learning brings to the fore is that of how learning identifies new action routines that are likely to provide useful building blocks in solving a wide range of future problems. Here and at many other points, hierarchical reinforcement learning offers an appealing framework for investigating the computational and neural underpinnings of hierarchically structured behavior.

Subfields: Cognitive Neuroscience

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

Citations in the corpus

Hierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic Motivation (2016, background): “ value function in terms of the expected future state occupancy. Decomposition of the successor representation yields reasonable sub-goals for spatial navigation problems [5, 12, 44]. Botvinick et al.[3] have written a general overview of hierarchical reinforcement learning in the context of cogniti”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “There is also significant evidence that substantial hierarchical structure is present in the human learning process (Botvinick, Niv, and Barto, 2009; Gershman, Pesaran, and Daw, 2009; Frank and Badre, 2012; Badre and Frank, 2012; Diuk, Schapiro, Córdova, Ribas-Fernandes, Niv, and Botvinick, 2013;…”
Automatically Composing Representation Transformations as a Means for Generalization (2018, positive): “Several works have investigated the conditions in which hierarchy is useful for humans [12, 69, 77]; our experiments show that the hierarchical structure of CRL is more useful than the ﬂat structure of the RNN for low-data compositional problems.”
Strategic Attentive Writer for Learning Macro-Actions (2016, background): “First and foremost, it facilitates structured exploration in reinforcement learning – as the network learns meaningful action patterns it can use them to make longer exploratory steps in the state space [4].”
Toward an Integration of Deep Learning and Neuroscience (2016, background): “There is evidence that the brain uses such a strategy (Botvinick et al., 2009; Botvinick and Weinstein, 2014), beside neural network demonstrations (Wayne and Abbott, 2014).”
SWIRL: A sequential windowed inverse reinforcement learning algorithm for robot tasks with delayed rewards (2018, background): “1999; Barto and Mahadevan 2003) in AI and in the analysis of biological systems (Botvinick 2008; Botvinick et al. 2009; Solway et al. 2014; Zacks et al. 2011; Whiten et al. 2006).”
Hierarchical models of behavior and prefrontal function. (2008, methodology): “(c) An actor–critic implementation of HRL, from Botvinick, Niv and Barto [ 27 ].”
Divide and Conquer: Hierarchical Reinforcement Learning and Task Decomposition in Humans (2013, methodology): “As reviewed by Botvinick et al. (2009), research suggests that representations of reward in OFC can be sensitive to shifts in response strategy or task set (O'Doherty et al., 2003; Schoenbaum et al., 1999), linking precisely with the idea that OFC might represent option-specific state values.”
The Role of Executive Function in Shaping Reinforcement Learning. (2020, background): “The theoretical framework of hierarchical RL also dissociates the role of exploiting information about the environment from the role of primary/secondary rewards, while emphasizing that both act as a teaching signal [63].”
Reinforcement learning in the brain (2009, background): “But, above and beyond the issue of learning of modules, the hierarchical RL framework raises many tantalizing questions regarding the neural implementation of hierarchical control (Botvinick, 2008; Botvinick et al., 2008).”
Design Principles of the Hippocampal Cognitive Map (2014, methodology): “Recent experimental work suggests that the brain may exploit a similar strategy [3, 36, 8].”
When Waiting is not an Option : Learning Options with a Deliberation Cost (2017, result): “Our assumption on the cost structure is also consistent with models of the prefrontal areas (Botvinick et al. 2009; Solway et al. 2014) presenting decision making over options as a slower model-based planning process as opposed to the fast and habitual learning taking place within an option.”
Reinforcement learning, efficient coding, and the statistics of natural tasks (2015, methodology): “This work drew on the computational framework of hierarchical reinforcement learning (HRL), a variation on RL in which the action set includes temporally extended behaviors or subroutines [5,32].”
The expected value of control: An integrative theory of anterior cingulate cortex function (2013, positive): “This account is theoretically motivated by hierarchical reinforcement learning (HRL; Botvinick et al., 2009b) and has found support in the recent finding that prediction error signals specifically anticipated by HRL are observed within dACC (Ribas-Fernandes et al., 2011).”