Optimal Behavioral Hierarchy

By Alec Solway, Carlos Diuk, Natalia Córdova, Debbie M. Yee, Andrew Barto, Yael Niv, Matthew Botvinick. 2014

173 citations recorded by the source corpus.

Human behavior has long been recognized to display hierarchical structure: actions fit together into subtasks, which cohere into extended goal-directed activities. Arranging actions hierarchically has well established benefits, allowing behaviors to be represented efficiently by the brain, and allowing solutions to new tasks to be discovered easily. However, these payoffs depend on the particular way in which actions are organized into a hierarchy, the specific way in which tasks are carved up into subtasks. We provide a mathematical account for what makes some hierarchies better than others, an account that allows an optimal hierarchy to be identified for any set of tasks. We then present results from four behavioral experiments, suggesting that human learners spontaneously discover optimal action hierarchies.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Developmental and Educational Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Developmental and Educational Psychology

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “There is also significant evidence that substantial hierarchical structure is present in the human learning process (Botvinick et al., 2009; Gershman et al., 2009; Frank and Badre, 2012; Badre and Frank, 2012; Diuk et al., 2013; Botvinick and Weinstein, 2014; Solway et al., 2014; Ribas-Fernandes et ”
Automatically Composing Representation Transformations as a Means for Generalization (2018, positive): “Several works have investigated the conditions in which hierarchy is useful for humans [12, 69, 77]; our experiments show that the hierarchical structure of CRL is more useful than the ﬂat structure of the RNN for low-data compositional problems.”
Multi-Level Discovery of Deep Options (2017, background): “Early work in hierarchical control demonstrated the advantages of hierarchical structures by handcrafting hierarchical policies [19] and by learning them given various manual speciﬁcations: state abstractions [20, 21, 22, 23], a set of waypoints [24], low-level skills [25, 26, 27], a set of…”
A Laplacian Framework for Option Discovery in Reinforcement Learning (2017, background): “Exploration, while traditionally a separate problem from option discovery, can also be addressed through the careful construction of options (McGovern & Barto, 2001; Şimşek et al., 2005; Solway et al., 2014; Kulkarni et al., 2016).”
SWIRL: A sequential windowed inverse reinforcement learning algorithm for robot tasks with delayed rewards (2018, background): “1999; Barto and Mahadevan 2003) in AI and in the analysis of biological systems (Botvinick 2008; Botvinick et al. 2009; Solway et al. 2014; Zacks et al. 2011; Whiten et al. 2006).”
A Geometric Perspective on Optimal Representations for Reinforcement Learning (2019, methodology): “We perform all of our experiments within the four-room domain (Sutton et al., 1999), one of the most commonly used environment for studying representation learning for RL (see e.g. Solway et al., 2014; Machado et al., 2017).”
Hierarchy Through Composition with Multitask LMDPs (2017, background): “Firstly, they are intuitive, as humans seldom plan at the level of raw actions, typically preferring to reason at a higher level of abstraction (Botvinick et al., 2009; Ribas-Fernandes et al., 2011; Solway et al., 2014).”
When Waiting is not an Option : Learning Options with a Deliberation Cost (2017, result): “Our assumption on the cost structure is also consistent with models of the prefrontal areas (Botvinick et al. 2009; Solway et al. 2014) presenting decision making over options as a slower model-based planning process as opposed to the fast and habitual learning taking place within an option.”
Eigenoption Discovery through the Deep Successor Representation (2017, unknown): “In the reinforcement learning framework, options are a well-known formalization of the notion of actions extended in time; and they have been shown to speed up learning and planning when appropriately deﬁned ( e.g. , Brunskill & Li, 2014; Guo et al., 2017; Solway et al., 2014).”
Model-based hierarchical reinforcement learning and human action control (2014, background): “[25].”