Hierarchical Control Using Networks Trained with Higher-Level Forward Models

By Greg Wayne, L. F. Abbott. 2014

21 citations recorded by the source corpus.

We propose and develop a hierarchical approach to network control of complex tasks. In this approach, a low-level controller directs the activity of a “plant,” the system that performs the task. However, the low-level controller may be able to solve only fairly simple problems involving the plant. To accomplish more complex tasks, we introduce a higher-level controller that controls the lower-level controller. We use this system to direct an articulated truck to a specified location through an environment filled with static or moving obstacles. The final system consists of networks that have memorized associations between the sensory data they receive and the commands they issue. These networks are trained on a set of optimal associations generated by minimizing cost functions. Cost function minimization requires predicting the consequences of sequences of commands, which is achieved by constructing forward models, including a model of the lower-level controller. The forward models and cost minimization are used only during training, allowing the trained networks to respond rapidly. In general, the hierarchical approach can be extended to larger numbers of levels, dividing complex tasks into more manageable subtasks. The optimization procedure and the construction of the forward models and controllers can be performed in similar ways at each level of the hierarchy, which allows the system to be modified to perform other tasks or to be extended for more complex tasks without retraining lower-levels.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

Dialog-based Language Learning (2016, methodology): “Finally, forward prediction models, which we make use of in this work, have been used for learning eye tracking [21], controlling robot arms [12] and vehicles [26], and action-conditional video prediction in atari games [16, 23].”
The Mechanics of n-Player Differentiable Games (2018, background): “Copyright 2018 by the author(s). optimization (Pfau & Vinyals, 2016), synthetic gradients (Jaderberg et al., 2017), hierarchical reinforcement learning (Wayne & Abbott, 2014; Vezhnevets et al., 2017), curiosity (Pathak et al., 2017), and imaginative agents (Racani`ere et al., 2017).”
Learning and Transfer of Modulated Locomotor Controllers (2016, background): “These temporal abstractions may be applied to motor control [5, 26, 29], may be transferred to new tasks [1, 2, 21], and may also incorporate information hiding principles [6, 14].”
Toward an Integration of Deep Learning and Neuroscience (2016, background): “There is evidence that the brain uses such a strategy (Botvinick et al., 2009; Botvinick and Weinstein, 2014), beside neural network demonstrations (Wayne and Abbott, 2014).”
Differentiable Game Mechanics (2019, background): “, 2017), hierarchical reinforcement learning (Wayne and Abbott, 2014; Vezhnevets et al., 2017), intrinsic curiosity (Pathak et al.”
Competitive Gradient Descent (2019, methodology): “…in machine learning have been using multi-agent optimization to design highly ﬂexible objective functions for reinforcement learning (Liu et al., 2016; Pfau and Vinyals, 2016; Pathak et al., 2017; Wayne and Abbott, 2014; Vezhnevets et al., 2017) and generative models (Good-fellow et al., 2014).”
Stable Opponent Shaping in Differentiable Games (2018, background): “…including intrinsic curiosity (Pathak et al., 2017), imaginative agents (Racani ` ere et al., 2017), synthetic gradients (Jaderberg et al., 2017), hierarchical reinforcement learning (RL) (Wayne & Abbott, 2014; Vezhnevets et al., 2017) and multi-agent RL in general (Busoniu et al., 2008).”